Re: [openstack-dev] [OPNFV] [Functest] Tempest & Rally
Morgan, You should add at least: sla: failure_rate: max: 0 Otherwise rally will pass 100% no matter what is happening. Best regards, Boris Pavlovic On Thu, Sep 24, 2015 at 10:29 PM, Tikkanen, Viktor (Nokia - FI/Espoo) < viktor.tikka...@nokia.com> wrote: > Hi Morgan > > and thank you for the overview. > > I'm now waiting for the POD#2 VPN profile (will be ready soon). We will > try then to figure out what OpenStack/tempest/rally configuration changes > are needed in order to get rid of those test failures. > > I suppose that most of the problems (like "Multiple possible networks > found" etc.) are relatively easy to solve. > > BTW, since tempest is being currently developed in "branchless" mode > (without release specific stable versions), do we have some common > understanding/requirements how "dynamically" Functest should use its code? > > For example, config_functest.py seems to contain routines for > cloning/installing rally (and indirectly tempest) code, does it mean that > the code will be cloned/installed at the time when the test set is executed > for the first time? (I'm just wondering if it is necessary or not to > "freeze" somehow used code for each OPNFV release to make sure that it will > remain compatible and that test results will be comparable between > different OPNFV setups). > > -Viktor > > > -Original Message- > > From: EXT morgan.richo...@orange.com [mailto:morgan.richo...@orange.com] > > Sent: Thursday, September 24, 2015 4:56 PM > > To: Kosonen, Juha (Nokia - FI/Espoo); Tikkanen, Viktor (Nokia - FI/Espoo) > > Cc: Jose Lausuch > > Subject: [OPNFV] [Functest] Tempest & Rally > > > > Hi, > > > > I was wondering whether you could have a look at Rally/Tempest tests we > > automatically launch in Functest. > > We have still some errors and I assume most of them are due to > > misconfiguration and/or quota ... > > With Jose, we planned to have a look after SR0 but we do not have much > > time and we are not fully skilled (even if we progressed a little bit:)) > > > > If you could have a look and give your feedback, it would be very > > helpful, we could discuss it during an IRC weekly meeting > > In Arno we did not use the SLA criteria, that is also something we could > > do for the B Release > > > > for instance if you look at > > https://build.opnfv.org/ci/view/functest/job/functest-foreman- > > master/19/consoleText > > > > you will see rally and Tempest log > > > > Rally scenario are a compilation of default Rally scenario played one > > after the other and can be found in > > > https://git.opnfv.org/cgit/functest/tree/testcases/VIM/OpenStack/CI/suites > > > > the Rally artifacts are also pushed into the artifact server > > http://artifacts.opnfv.org/ > > e.g. > > http://artifacts.opnfv.org/functest/lf_pod2/2015-09-23_17-36- > > 07/results/rally/opnfv-authenticate.html > > look for 09-23 to get Rally json/html files and tempest.conf > > > > thanks > > > > Morgan > > > > > > > __ > > ___ > > > > Ce message et ses pieces jointes peuvent contenir des informations > > confidentielles ou privilegiees et ne doivent donc > > pas etre diffuses, exploites ou copies sans autorisation. Si vous avez > > recu ce message par erreur, veuillez le signaler > > a l'expediteur et le detruire ainsi que les pieces jointes. Les messages > > electroniques etant susceptibles d'alteration, > > Orange decline toute responsabilite si ce message a ete altere, deforme > ou > > falsifie. Merci. > > > > This message and its attachments may contain confidential or privileged > > information that may be protected by law; > > they should not be distributed, used or copied without authorisation. > > If you have received this email in error, please notify the sender and > > delete this message and its attachments. > > As emails may be altered, Orange is not liable for messages that have > been > > modified, changed or falsified. > > Thank you. > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [OPNFV] [Functest] Tempest & Rally
Jose, Rally community provides official docker images here: https://hub.docker.com/r/rallyforge/rally/ So I would suggest to use them. Best regards, Boris Pavlovic On Fri, Sep 25, 2015 at 5:07 AM, Jose Lausuch <jose.laus...@ericsson.com> wrote: > Hi, > > > > Thanks for the hint Boris. > > > > Regarding what we do at functest with Rally, yes, we clone the latest from > the Rally repo. We thought about that before and the possible errors it can > convey, compatibility and so on. > > > > As I am working on a Docker image where all the Functest environment will > be pre-installed, we might get rid of such potential problems. But, that > image will need constant updates if there are major patches/bugfixes in the > rally repo. > > > > What is your opinion on this? What do you think it makes more sense? > > > > /Jose > > > > > > > > *From:* bo...@pavlovic.ru [mailto:bo...@pavlovic.ru] *On Behalf Of *Boris > Pavlovic > *Sent:* Friday, September 25, 2015 7:56 AM > *To:* OpenStack Development Mailing List (not for usage questions) > *Cc:* EXT morgan.richo...@orange.com; Kosonen, Juha (Nokia - FI/Espoo); > Jose Lausuch > *Subject:* Re: [openstack-dev] [OPNFV] [Functest] Tempest & Rally > > > > Morgan, > > > > > > You should add at least: > > > > sla: > failure_rate: > max: 0 > > > > Otherwise rally will pass 100% no matter what is happening. > > > > > > Best regards, > > Boris Pavlovic > > > > On Thu, Sep 24, 2015 at 10:29 PM, Tikkanen, Viktor (Nokia - FI/Espoo) < > viktor.tikka...@nokia.com> wrote: > > Hi Morgan > > and thank you for the overview. > > I'm now waiting for the POD#2 VPN profile (will be ready soon). We will > try then to figure out what OpenStack/tempest/rally configuration changes > are needed in order to get rid of those test failures. > > I suppose that most of the problems (like "Multiple possible networks > found" etc.) are relatively easy to solve. > > BTW, since tempest is being currently developed in "branchless" mode > (without release specific stable versions), do we have some common > understanding/requirements how "dynamically" Functest should use its code? > > For example, config_functest.py seems to contain routines for > cloning/installing rally (and indirectly tempest) code, does it mean that > the code will be cloned/installed at the time when the test set is executed > for the first time? (I'm just wondering if it is necessary or not to > "freeze" somehow used code for each OPNFV release to make sure that it will > remain compatible and that test results will be comparable between > different OPNFV setups). > > -Viktor > > > -Original Message- > > From: EXT morgan.richo...@orange.com [mailto:morgan.richo...@orange.com] > > Sent: Thursday, September 24, 2015 4:56 PM > > To: Kosonen, Juha (Nokia - FI/Espoo); Tikkanen, Viktor (Nokia - FI/Espoo) > > Cc: Jose Lausuch > > Subject: [OPNFV] [Functest] Tempest & Rally > > > > Hi, > > > > I was wondering whether you could have a look at Rally/Tempest tests we > > automatically launch in Functest. > > We have still some errors and I assume most of them are due to > > misconfiguration and/or quota ... > > With Jose, we planned to have a look after SR0 but we do not have much > > time and we are not fully skilled (even if we progressed a little bit:)) > > > > If you could have a look and give your feedback, it would be very > > helpful, we could discuss it during an IRC weekly meeting > > In Arno we did not use the SLA criteria, that is also something we could > > do for the B Release > > > > for instance if you look at > > https://build.opnfv.org/ci/view/functest/job/functest-foreman- > > master/19/consoleText > > > > you will see rally and Tempest log > > > > Rally scenario are a compilation of default Rally scenario played one > > after the other and can be found in > > > https://git.opnfv.org/cgit/functest/tree/testcases/VIM/OpenStack/CI/suites > > > > the Rally artifacts are also pushed into the artifact server > > http://artifacts.opnfv.org/ > > e.g. > > http://artifacts.opnfv.org/functest/lf_pod2/2015-09-23_17-36- > > 07/results/rally/opnfv-authenticate.html > > look for 09-23 to get Rally json/html files and tempest.conf > > > > thanks > > > > Morgan > > > > > > > __ > >
Re: [openstack-dev] [OPNFV] [Functest] Tempest & Rally
Hi Morgan and thank you for the overview. I'm now waiting for the POD#2 VPN profile (will be ready soon). We will try then to figure out what OpenStack/tempest/rally configuration changes are needed in order to get rid of those test failures. I suppose that most of the problems (like "Multiple possible networks found" etc.) are relatively easy to solve. BTW, since tempest is being currently developed in "branchless" mode (without release specific stable versions), do we have some common understanding/requirements how "dynamically" Functest should use its code? For example, config_functest.py seems to contain routines for cloning/installing rally (and indirectly tempest) code, does it mean that the code will be cloned/installed at the time when the test set is executed for the first time? (I'm just wondering if it is necessary or not to "freeze" somehow used code for each OPNFV release to make sure that it will remain compatible and that test results will be comparable between different OPNFV setups). -Viktor > -Original Message- > From: EXT morgan.richo...@orange.com [mailto:morgan.richo...@orange.com] > Sent: Thursday, September 24, 2015 4:56 PM > To: Kosonen, Juha (Nokia - FI/Espoo); Tikkanen, Viktor (Nokia - FI/Espoo) > Cc: Jose Lausuch > Subject: [OPNFV] [Functest] Tempest & Rally > > Hi, > > I was wondering whether you could have a look at Rally/Tempest tests we > automatically launch in Functest. > We have still some errors and I assume most of them are due to > misconfiguration and/or quota ... > With Jose, we planned to have a look after SR0 but we do not have much > time and we are not fully skilled (even if we progressed a little bit:)) > > If you could have a look and give your feedback, it would be very > helpful, we could discuss it during an IRC weekly meeting > In Arno we did not use the SLA criteria, that is also something we could > do for the B Release > > for instance if you look at > https://build.opnfv.org/ci/view/functest/job/functest-foreman- > master/19/consoleText > > you will see rally and Tempest log > > Rally scenario are a compilation of default Rally scenario played one > after the other and can be found in > https://git.opnfv.org/cgit/functest/tree/testcases/VIM/OpenStack/CI/suites > > the Rally artifacts are also pushed into the artifact server > http://artifacts.opnfv.org/ > e.g. > http://artifacts.opnfv.org/functest/lf_pod2/2015-09-23_17-36- > 07/results/rally/opnfv-authenticate.html > look for 09-23 to get Rally json/html files and tempest.conf > > thanks > > Morgan > > > __ > ___ > > Ce message et ses pieces jointes peuvent contenir des informations > confidentielles ou privilegiees et ne doivent donc > pas etre diffuses, exploites ou copies sans autorisation. Si vous avez > recu ce message par erreur, veuillez le signaler > a l'expediteur et le detruire ainsi que les pieces jointes. Les messages > electroniques etant susceptibles d'alteration, > Orange decline toute responsabilite si ce message a ete altere, deforme ou > falsifie. Merci. > > This message and its attachments may contain confidential or privileged > information that may be protected by law; > they should not be distributed, used or copied without authorisation. > If you have received this email in error, please notify the sender and > delete this message and its attachments. > As emails may be altered, Orange is not liable for messages that have been > modified, changed or falsified. > Thank you. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
