Changing subject line to continue thread about new $subj
On 08/12/2014 08:56 AM, Doug Hellmann wrote:
On Aug 11, 2014, at 12:00 PM, David Kranz <dkr...@redhat.com
<mailto:dkr...@redhat.com>> wrote:
On 08/06/2014 05:48 PM, John Griffith wrote:
I have to agree with Duncan here. I also don't know if I fully
understand the limit in options. Stress test seems like it
could/should be different (again overlap isn't a horrible thing) and
I don't see it as siphoning off resources so not sure of the issue.
We've become quite wrapped up in projects, programs and the like
lately and it seems to hinder forward progress more than anything else.
I'm also not convinced that Tempest is where all things belong, in
fact I've been thinking more and more that a good bit of what
Tempest does today should fall more on the responsibility of the
projects themselves. For example functional testing of features
etc, ideally I'd love to have more of that fall on the projects and
their respective teams. That might even be something as simple to
start as saying "if you contribute a new feature, you have to also
provide a link to a contribution to the Tempest test-suite that
checks it". Sort of like we do for unit tests, cross-project
tracking is difficult of course, but it's a start. The other idea
is maybe functional test harnesses live in their respective projects.
Honestly I think who better to write tests for a project than the
folks building and contributing to the project. At some point IMO
the QA team isn't going to scale. I wonder if maybe we should be
thinking about proposals for delineating responsibility and goals in
terms of functional testing?
All good points. Your last paragraph was discussed by the QA team
leading up to and at the Atlanta summit. The conclusion was that the
api/functional tests focused on a single project should be part of
that project. As Sean said, we can envision there being half (or some
other much smaller number) as many such tests in tempest going forward.
Details are under discussion, but the way this is likely to play out
is that individual projects will start by creating their own
functional tests outside of tempest. Swift already does this and
neutron seems to be moving in that direction. There is a spec to
break out parts of tempest
(https://github.com/openstack/qa-specs/blob/master/specs/tempest-library.rst)
into a library that might be used by projects implementing functional
tests.
Once a project has "sufficient" functional testing, we can consider
removing its api tests from tempest. This is a bit tricky because
tempest needs to cover *all* cross-project interactions. In this
respect, there is no clear line in tempest between scenario tests
which have this goal explicitly, and api tests which may also involve
interactions that might not be covered in a scenario. So we will need
a principled way to make sure there is complete cross-project
coverage in tempest with a smaller number of api tests.
-David
We need to be careful about dumping the tests from tempest now that
the DefCore group is relying on them as well. Tempest is no longer
just a developer/QA/operations tool. It's also being used as the basis
of a trademark enforcement tool. That's not to say we can't change the
test suite, but we have to consider a new angle when doing so.
Doug
Thanks, Doug. We need to get away from "acceptance test == tempest"
while retaining the ability to define and run an acceptance test as
easily as tempest can be run now. My view is that functional tests in
projects should have the capability to be run against real clouds, and
that in-project functional tests should "look like", and be
interchangeable with, the api tests in tempest. The in-project tests
would be focused on completeness of api testing and the tempest tests
focused on cross-project interaction, but they could be run in similar
ways. Then a trademark enforcement tool, or any other kind of acceptance
test, could select which tests to run. I think this view may be a bit
controversial but your point obviously needs to be addressed.
-David
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
<mailto:OpenStack-dev@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
