Sorry for wrong numbers. The bug-fix for issue with counters is merged. Correct numbers(latest result from rally's gate[1]): - total number of executed tests: 1689 - success: 1155 - skipped: 534 (neutron,heat,sahara,ceilometer are disabled. [2] should enable them) - failed: 0
[1] - http://logs.openstack.org/27/246627/11/gate/gate-rally-dsvm-verify-full/800bad0/rally-verify/7_verify_results_--html.html.gz [2] - https://review.openstack.org/#/c/250540/ On Thu, Nov 26, 2015 at 3:23 PM, Yaroslav Lobankov <[email protected]> wrote: > Hello everyone, > > Yes, I am working on this now. We have some success already, but there is > a lot of work to do. Of course, some things don't work ideally. For > example, in [2] from the previous letter we have not 24 skipped tests, > actually much more. So we have a bug somewhere :) > > Regards, > Yaroslav Lobankov. > > On Thu, Nov 26, 2015 at 3:59 PM, Andrey Kurilin <[email protected]> > wrote: > >> Hi! >> Boris P. and I tried to push a spec[1] for automation tempest config >> generator, but we did not succeed to merge it. Imo, qa-team doesn't want to >> have such tool:( >> >> >However, there is a big concern: >> >If the script contain a bug and creates the configuration which makes >> >most tests skipped, we cannot do enough tests on the gate. >> >Tempest contains 1432 tests and difficult to detect which tests are >> >skipped as unexpected. >> >> Yaroslav Lobankov is working on improvement for tempest config generator >> in Rally. Last time when we launch full tempest run[2], we got 1154 success >> tests and only 24 skipped. Also, there is a patch, which adds x-fail >> mechanism(it based on subunit-filter): you can transmit a file with test >> names + reasons and rally will modify results. >> >> [1] - https://review.openstack.org/#/c/94473/ >> >> [2] - >> http://logs.openstack.org/49/242849/8/check/gate-rally-dsvm-verify/e91992e/rally-verify/7_verify_results_--html.html.gz >> >> On Thu, Nov 26, 2015 at 1:52 PM, Ken'ichi Ohmichi <[email protected]> >> wrote: >> >>> Hi Daniel, >>> >>> Thanks for pointing this up. >>> >>> 2015-11-25 1:40 GMT+09:00 Daniel Mellado <[email protected]>: >>> > Hi All, >>> > >>> > As you might already know, within Red Hat's tempest fork, we do have >>> one >>> > tempest configuration script which was built in the past by David >>> Kranz [1] >>> > and that's been actively used in our CI system. Regarding this topic, >>> I'm >>> > aware that quite some effort has been done in the past [2] and I would >>> like >>> > to complete the implementation of this blueprint/spec. >>> > >>> > My plan would be to have this script under the /tempest/cmd or >>> > /tempest/tools folder from tempest so it can be used to configure not >>> the >>> > tempest gate but any cloud we'd like to run tempest against. >>> > >>> > Adding the configuration script was discussed briefly at the Mitaka >>> summit >>> > in the QA Priorities meting [3]. I propose we use the existing >>> etherpad to >>> > continue the discussion around and tracking of implementing "tempest >>> > config-create" using the downstream config script as a starting point. >>> [4] >>> > >>> > If you have any questions, comments or opinion, please let me know. >>> >>> This topic have happened several times, and I also felt this kind of >>> tool was very useful for Tempest users, because Tempest contains 296 >>> options($ grep cfg * -R | grep Opt | wc -l) now and it is difficult to >>> set the configuration up. >>> However, there is a big concern: >>> If the script contain a bug and creates the configuration which makes >>> most tests skipped, we cannot do enough tests on the gate. >>> Tempest contains 1432 tests and difficult to detect which tests are >>> skipped as unexpected. >>> Actually we faced unexpected skipped tests on the gate before due to >>> some bug, then the problem has been fixed. >>> But I can imagine this kind of problem happens after implementing this >>> kind of script. >>> >>> So now I am feeling Tempest users need to know what cloud they want to >>> test with Tempest, and need to know what tests run with Tempest. >>> Testers need to know what test target/items they are testing basically. >>> >>> Thanks >>> Ken Ohmichi >>> >>> --- >>> >>> > --- >>> > [1] >>> > >>> https://github.com/redhat-openstack/tempest/blob/master/tools/config_tempest.py >>> > [2] >>> https://blueprints.launchpad.net/tempest/+spec/tempest-config-generator >>> > [3] https://etherpad.openstack.org/p/mitaka-qa-priorities >>> > [4] https://etherpad.openstack.org/p/tempest-cli-improvements >>> > >>> > >>> https://github.com/openstack/qa-specs/blob/master/specs/tempest/tempest-cli-improvements.rst >>> > >>> > >>> __________________________________________________________________________ >>> > OpenStack Development Mailing List (not for usage questions) >>> > Unsubscribe: >>> [email protected]?subject:unsubscribe >>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> > >>> >>> >>> __________________________________________________________________________ >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: >>> [email protected]?subject:unsubscribe >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >> >> >> >> -- >> Best regards, >> Andrey Kurilin. >> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> [email protected]?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: [email protected]?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > -- Best regards, Andrey Kurilin.
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
