Re: [openstack-dev] [qa][gate] tempest slow - where do we execute them in gate?

[Matt Riedemann]

On 4/17/2017 1:55 PM, Ihar Hrachyshka wrote:

But since it's not executed anywhere in tempest gate, even as
non-voting (?), it's effectively dead code that may be long broken
without anyone knowing. Of course there are consumers of the tests
downstream, but for those consumers it's a tough call to start
depending on the tests if they are not sanity checked by tempest
itself. Wouldn't it make sense to have some job in tempest gate that
would execute those tests (maybe just them to speed up such a job?
maybe non-voting? maybe even as periodic? but there should be
something that keeps it green in long run).


I enabled the job that runs the slow scenario tests in the nova 
experimental queue:


https://review.openstack.org/#/c/458676/

Because we were changing some of the encrypted volume code in nova and 
that's only exercised by a scenario test that was marked slow, but as 
you said I need to run tests to have coverage on those changes. That 
works for me in this case, the problem is I had to (1) know those tests 
existed and (2) go out of my way looking for their results in a job run 
and find out they weren't there, then hunt them out. That's exceptional 
for most patches so yes, things are going to slip through and break most 
likely because we're not gating on them. I understand why the QA team 
did what they did though so I'm not pushing back on that.


--

Thanks,

Matt

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][gate] tempest slow - where do we execute them in gate?

[Andrea Frittoli]

On Mon, Apr 17, 2017 at 7:58 PM Ihar Hrachyshka  wrote:

> On Mon, Apr 17, 2017 at 9:35 AM, Jordan Pittier
>  wrote:
> > We don"t run slow tests because the QA team think that they don't bring
> > enough value to be executed, every time and everywhere. The idea is that
> if
> > some specific slow tests are of some interest to some specific openstack
> > projects, those projects can change the config of their jobs to enable
> these
> > tests.
>
>
> But since it's not executed anywhere in tempest gate, even as
> non-voting (?), it's effectively dead code that may be long broken
> without anyone knowing. Of course there are consumers of the tests
> downstream, but for those consumers it's a tough call to start
> depending on the tests if they are not sanity checked by tempest
> itself. Wouldn't it make sense to have some job in tempest gate that
> would execute those tests (maybe just them to speed up such a job?
> maybe non-voting?


We reduced the number of scenario tests we run in the main gate because
some of those tests are not so
critical for the integrated gate, and also because we had an unstable gate
we had to deal with to allow for
Pike development to happen.

The SUT was overloaded in the gate, due to a combination of effects: too
many services running, many
consuming more memory that it did in the past, as well as too many "heavy"
tests running in parallel.

As soon as removed scenario tests from the main gate we introduced a new
non-voting job against
Tempest, which runs *all* scenario tests against a multimode environment,
see for instance [0], so
we can ensure the code is not dead as long as it lives in Tempest.

I'm planning to add more "heavy/long" tests to that jobs, e.g. compute
migrate ones [1].

andreaf

[0]
http://logs.openstack.org/69/451769/2/check/gate-tempest-dsvm-neutron-scenario-multinode-ubuntu-xenial-nv/1f71416/

[1] https://review.openstack.org/#/c/451769/


> maybe even as periodic? but there should be
> something that keeps it green in long run).
>
> Ihar
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][gate] tempest slow - where do we execute them in gate?

[Matthew Treinish]

On Mon, Apr 17, 2017 at 11:55:28AM -0700, Ihar Hrachyshka wrote:
> On Mon, Apr 17, 2017 at 9:35 AM, Jordan Pittier
>  wrote:
> > We don"t run slow tests because the QA team think that they don't bring
> > enough value to be executed, every time and everywhere. The idea is that if
> > some specific slow tests are of some interest to some specific openstack
> > projects, those projects can change the config of their jobs to enable these
> > tests.
> 
> 
> But since it's not executed anywhere in tempest gate, even as
> non-voting (?), it's effectively dead code that may be long broken
> without anyone knowing. Of course there are consumers of the tests
> downstream, but for those consumers it's a tough call to start
> depending on the tests if they are not sanity checked by tempest
> itself. Wouldn't it make sense to have some job in tempest gate that
> would execute those tests (maybe just them to speed up such a job?
> maybe non-voting? maybe even as periodic? but there should be
> something that keeps it green in long run).
> 

In theory those tests are already supposed to be run as a periodic/experimental
job. The periodic-tempest-dsvm-all-master is setup to run all tests, including
those tagged as slow. However, the job has been broken for some time, I didn't
even notice until I looked just now. (openstack-health didn't show it because
it fails before subunit is generated) I'll pushed:

https://review.openstack.org/#/c/457334/

to fix the job. Once that lands lets see how far things have bitrotted in there.

-Matt Treinish


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][gate] tempest slow - where do we execute them in gate?

[Ihar Hrachyshka]

On Mon, Apr 17, 2017 at 9:35 AM, Jordan Pittier
 wrote:
> We don"t run slow tests because the QA team think that they don't bring
> enough value to be executed, every time and everywhere. The idea is that if
> some specific slow tests are of some interest to some specific openstack
> projects, those projects can change the config of their jobs to enable these
> tests.


But since it's not executed anywhere in tempest gate, even as
non-voting (?), it's effectively dead code that may be long broken
without anyone knowing. Of course there are consumers of the tests
downstream, but for those consumers it's a tough call to start
depending on the tests if they are not sanity checked by tempest
itself. Wouldn't it make sense to have some job in tempest gate that
would execute those tests (maybe just them to speed up such a job?
maybe non-voting? maybe even as periodic? but there should be
something that keeps it green in long run).

Ihar

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][gate] tempest slow - where do we execute them in gate?

[Jordan Pittier]

On Mon, Apr 17, 2017 at 6:50 AM, Ihar Hrachyshka 
wrote:

> Hi all,
>
> so I tried to inject a failure in a tempest test and was surprised
> that no gate job failed because of that:
> https://review.openstack.org/#/c/457102/1
>
> It turned out that the test is not executed because we always ignore
> all 'slow' tagged test cases:
> http://logs.openstack.org/02/457102/1/check/gate-tempest-
> dsvm-neutron-full-ubuntu-xenial/89a08cc/console.html#_
> 2017-04-17_01_43_39_115768

Indeed, we don't run slow tests. Many network scenarios are not run since
https://review.openstack.org/#/c/439698/


>
>
> Question: do we execute those tests anywhere in gate, and if so,
> where? (And if not, why, and how do we guarantee that they are not
> broken by new changes?)
>
We don"t run slow tests because the QA team think that they don't bring
enough value to be executed, every time and everywhere. The idea is that if
some specific slow tests are of some interest to some specific openstack
projects, those projects can change the config of their jobs to enable
these tests.



>
> Ihar
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [qa][gate] tempest slow - where do we execute them in gate?

[Ihar Hrachyshka]

Hi all,

so I tried to inject a failure in a tempest test and was surprised
that no gate job failed because of that:
https://review.openstack.org/#/c/457102/1

It turned out that the test is not executed because we always ignore
all 'slow' tagged test cases:
http://logs.openstack.org/02/457102/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/89a08cc/console.html#_2017-04-17_01_43_39_115768

Question: do we execute those tests anywhere in gate, and if so,
where? (And if not, why, and how do we guarantee that they are not
broken by new changes?)

Ihar

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev