Re: [openstack-dev] [qa][gate] tempest slow - where do we execute them in gate?
On 4/17/2017 1:55 PM, Ihar Hrachyshka wrote: But since it's not executed anywhere in tempest gate, even as non-voting (?), it's effectively dead code that may be long broken without anyone knowing. Of course there are consumers of the tests downstream, but for those consumers it's a tough call to start depending on the tests if they are not sanity checked by tempest itself. Wouldn't it make sense to have some job in tempest gate that would execute those tests (maybe just them to speed up such a job? maybe non-voting? maybe even as periodic? but there should be something that keeps it green in long run). I enabled the job that runs the slow scenario tests in the nova experimental queue: https://review.openstack.org/#/c/458676/ Because we were changing some of the encrypted volume code in nova and that's only exercised by a scenario test that was marked slow, but as you said I need to run tests to have coverage on those changes. That works for me in this case, the problem is I had to (1) know those tests existed and (2) go out of my way looking for their results in a job run and find out they weren't there, then hunt them out. That's exceptional for most patches so yes, things are going to slip through and break most likely because we're not gating on them. I understand why the QA team did what they did though so I'm not pushing back on that. -- Thanks, Matt __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][gate] tempest slow - where do we execute them in gate?
On Mon, Apr 17, 2017 at 7:58 PM Ihar Hrachyshka wrote: > On Mon, Apr 17, 2017 at 9:35 AM, Jordan Pittier > wrote: > > We don"t run slow tests because the QA team think that they don't bring > > enough value to be executed, every time and everywhere. The idea is that > if > > some specific slow tests are of some interest to some specific openstack > > projects, those projects can change the config of their jobs to enable > these > > tests. > > > But since it's not executed anywhere in tempest gate, even as > non-voting (?), it's effectively dead code that may be long broken > without anyone knowing. Of course there are consumers of the tests > downstream, but for those consumers it's a tough call to start > depending on the tests if they are not sanity checked by tempest > itself. Wouldn't it make sense to have some job in tempest gate that > would execute those tests (maybe just them to speed up such a job? > maybe non-voting? We reduced the number of scenario tests we run in the main gate because some of those tests are not so critical for the integrated gate, and also because we had an unstable gate we had to deal with to allow for Pike development to happen. The SUT was overloaded in the gate, due to a combination of effects: too many services running, many consuming more memory that it did in the past, as well as too many "heavy" tests running in parallel. As soon as removed scenario tests from the main gate we introduced a new non-voting job against Tempest, which runs *all* scenario tests against a multimode environment, see for instance [0], so we can ensure the code is not dead as long as it lives in Tempest. I'm planning to add more "heavy/long" tests to that jobs, e.g. compute migrate ones [1]. andreaf [0] http://logs.openstack.org/69/451769/2/check/gate-tempest-dsvm-neutron-scenario-multinode-ubuntu-xenial-nv/1f71416/ [1] https://review.openstack.org/#/c/451769/ > maybe even as periodic? but there should be > something that keeps it green in long run). > > Ihar > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][gate] tempest slow - where do we execute them in gate?
On Mon, Apr 17, 2017 at 11:55:28AM -0700, Ihar Hrachyshka wrote: > On Mon, Apr 17, 2017 at 9:35 AM, Jordan Pittier > wrote: > > We don"t run slow tests because the QA team think that they don't bring > > enough value to be executed, every time and everywhere. The idea is that if > > some specific slow tests are of some interest to some specific openstack > > projects, those projects can change the config of their jobs to enable these > > tests. > > > But since it's not executed anywhere in tempest gate, even as > non-voting (?), it's effectively dead code that may be long broken > without anyone knowing. Of course there are consumers of the tests > downstream, but for those consumers it's a tough call to start > depending on the tests if they are not sanity checked by tempest > itself. Wouldn't it make sense to have some job in tempest gate that > would execute those tests (maybe just them to speed up such a job? > maybe non-voting? maybe even as periodic? but there should be > something that keeps it green in long run). > In theory those tests are already supposed to be run as a periodic/experimental job. The periodic-tempest-dsvm-all-master is setup to run all tests, including those tagged as slow. However, the job has been broken for some time, I didn't even notice until I looked just now. (openstack-health didn't show it because it fails before subunit is generated) I'll pushed: https://review.openstack.org/#/c/457334/ to fix the job. Once that lands lets see how far things have bitrotted in there. -Matt Treinish signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][gate] tempest slow - where do we execute them in gate?
On Mon, Apr 17, 2017 at 9:35 AM, Jordan Pittier wrote: > We don"t run slow tests because the QA team think that they don't bring > enough value to be executed, every time and everywhere. The idea is that if > some specific slow tests are of some interest to some specific openstack > projects, those projects can change the config of their jobs to enable these > tests. But since it's not executed anywhere in tempest gate, even as non-voting (?), it's effectively dead code that may be long broken without anyone knowing. Of course there are consumers of the tests downstream, but for those consumers it's a tough call to start depending on the tests if they are not sanity checked by tempest itself. Wouldn't it make sense to have some job in tempest gate that would execute those tests (maybe just them to speed up such a job? maybe non-voting? maybe even as periodic? but there should be something that keeps it green in long run). Ihar __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][gate] tempest slow - where do we execute them in gate?
On Mon, Apr 17, 2017 at 6:50 AM, Ihar Hrachyshka wrote: > Hi all, > > so I tried to inject a failure in a tempest test and was surprised > that no gate job failed because of that: > https://review.openstack.org/#/c/457102/1 > > It turned out that the test is not executed because we always ignore > all 'slow' tagged test cases: > http://logs.openstack.org/02/457102/1/check/gate-tempest- > dsvm-neutron-full-ubuntu-xenial/89a08cc/console.html#_ > 2017-04-17_01_43_39_115768 Indeed, we don't run slow tests. Many network scenarios are not run since https://review.openstack.org/#/c/439698/ > > > Question: do we execute those tests anywhere in gate, and if so, > where? (And if not, why, and how do we guarantee that they are not > broken by new changes?) > We don"t run slow tests because the QA team think that they don't bring enough value to be executed, every time and everywhere. The idea is that if some specific slow tests are of some interest to some specific openstack projects, those projects can change the config of their jobs to enable these tests. > > Ihar > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [qa][gate] tempest slow - where do we execute them in gate?
Hi all, so I tried to inject a failure in a tempest test and was surprised that no gate job failed because of that: https://review.openstack.org/#/c/457102/1 It turned out that the test is not executed because we always ignore all 'slow' tagged test cases: http://logs.openstack.org/02/457102/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/89a08cc/console.html#_2017-04-17_01_43_39_115768 Question: do we execute those tests anywhere in gate, and if so, where? (And if not, why, and how do we guarantee that they are not broken by new changes?) Ihar __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev