On 6/12/2014 10:51 AM, Matthew Treinish wrote:
On Fri, Jun 13, 2014 at 12:41:19AM +0930, Christopher Yeoh wrote:
On Fri, Jun 13, 2014 at 12:25 AM, Dan Smith <[email protected]> wrote:
I think it'd be OK to move them to the experimental queue and a periodic
nightly job until the v2.1 stuff shakes out. The v3 API is marked
experimental right now so it seems fitting that it'd be running tests in
the experimental queue until at least the spec is approved and
microversioning starts happening in the code base.
I think this is reasonable. Continuing to run the full set of tests on
every patch for something we never expect to see the light of day (in its
current form) seems wasteful to me. Plus, we're going to (presumably) be
ramping up tests on v2.1, which means to me that we'll need to clear out
some capacity to make room for that.
Thats true, though I was suggesting as v2.1microversions rolls out we drop
the test out of v3 and move it to v2.1microversions testing, so there's no
change in capacity required.
That's why I wasn't proposing that we rip the tests out of the tree. I'm just
trying to weigh the benefit of leaving them enabled on every run against
the increased load they cause in an arguably overworked gate.
Matt - how much of the time overhead is scenario tests? That's something
that would have a lot less impact if moved to and experimental queue.
Although the v3 api as a whole won't be officially exposed, the api tests
test specific features fairly indepdently which are slated for
v2.1microversions on a case by case basis and I don't want to see those
regress. I guess my concern is how often the experimental queue results get
really looked at and how hard/quick it is to revert when lots of stuff
merges in a short period of time)
The scenario tests tend to be the slower tests in tempest. I have to disagree
that removing them would have lower impact. The scenario tests provide the best
functional verification, which is part of the reason we always have failures in
the gate on them. While it would make the gate faster the decrease in what were
testing isn't worth it. Also, for reference I pulled the test run times that
were greater than 10sec out of a recent gate run:
http://paste.openstack.org/show/83827/
The experimental jobs aren't automatically run, they have to be manually
triggered by leaving a 'check experimental' comment. So for changes that we want
to test the v3 api on a comment would have to left. To prevent regression is why
we'd also have the nightly job, which I think is a better compromise for the v3
tests while we wait to migrate them to the v2.1 microversion tests.
Another, option is that we make the v3 job run only on the check queue and not
on the gate. But the benefits of that are slightly more limited, because we'd
still be holding up the check queue.
-Matt Treinish
_______________________________________________
OpenStack-dev mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Yeah the scenario tests need to stay, that's how we've exposed the two
big ssh bugs in the last couple of weeks which are obvious issues at scale.
I still think experimental/periodic is the way to go, not a hybrid of
check-on/gate-off. If we want to explicitly test v3 API changes we can
do that with 'recheck experimental'. Granted someone has to remember to
run those, much like checking/rechecking 3rd party CI results.
One issue I've had with the nightly periodic job is finding out where
the results are in an easy to consume format. Is there something out
there for that? I'm thinking specifically of things we've turned off in
the gate before like multi-backend volume tests and
allow_tenant_isolation=False.
--
Thanks,
Matt Riedemann
_______________________________________________
OpenStack-dev mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
