On 04/09/14 10:45, Jay Pipes wrote:
On 08/29/2014 05:15 PM, Zane Bitter wrote:
On 29/08/14 14:27, Jay Pipes wrote:
On 08/26/2014 10:14 AM, Zane Bitter wrote:
Steve Baker has started the process of moving Heat tests out of the
Tempest repository and into the Heat repository, and we're looking for
some guidance on how they should be packaged in a consistent way.
Apparently there are a few projects already packaging functional tests
in the package <projectname>.tests.functional (alongside
<projectname>.tests.unit for the unit tests).

That strikes me as odd in our context, because while the unit tests run
against the code in the package in which they are embedded, the
functional tests run against some entirely different code - whatever
OpenStack cloud you give it the auth URL and credentials for. So these
tests run from the outside, just like their ancestors in Tempest do.

There's all kinds of potential confusion here for users and packagers.
None of it is fatal and all of it can be worked around, but if we
refrain from doing the thing that makes zero conceptual sense then
will be no problem to work around :)

I suspect from reading the previous thread about "In-tree functional
test vision" that we may actually be dealing with three categories of
test here rather than two:

* Unit tests that run against the package they are embedded in
* Functional tests that run against the package they are embedded in
* Integration tests that run against a specified cloud

i.e. the tests we are now trying to add to Heat might be qualitatively
different from the <projectname>.tests.functional suites that already
exist in a few projects. Perhaps someone from Neutron and/or Swift can

I'd like to propose that tests of the third type get their own
package with a name of the form <projectname>-integrationtests (second
choice: <projectname>-tempest on the principle that they're essentially
plugins for Tempest). How would people feel about standardising that
across OpenStack?

By its nature, Heat is one of the only projects that would have
integration tests of this nature. For Nova, there are some "functional"
tests in nova/tests/integrated/ (yeah, badly named, I know) that are
tests of the REST API endpoints and running service daemons (the things
that are RPC endpoints), with a bunch of stuff faked out (like RPC
comms, image services, authentication and the hypervisor layer itself).
So, the "integrated" tests in Nova are really not testing integration
with other projects, but rather integration of the subsystems and
processes inside Nova.

I'd support a policy that true integration tests -- tests that test the
interaction between multiple real OpenStack service endpoints -- be left
entirely to Tempest. Functional tests that test interaction between
internal daemons and processes to a project should go into

For Heat, I believe tests that rely on faked-out other OpenStack
services but stress the interaction between internal Heat
daemons/processes should be in /heat/tests/functional/ and any tests the
rely on working, real OpenStack service endpoints should be in Tempest.

Well, the problem with that is that last time I checked there was
exactly one Heat scenario test in Tempest because tempest-core doesn't
have the bandwidth to merge all (any?) of the other ones folks submitted.

So we're moving them to openstack/heat for the pure practical reason
that it's the only way to get test coverage at all, rather than concerns
about overloading the gate or theories about the best venue for
cross-project integration testing.

Hmm, speaking of passive aggressivity...

That's probably a fair criticism, in light of what Matt said about the failures of communication on both sides. I think a formal liaison program will be an enormous help here. However, it won't change the fact that keeping the tests for every project in a single repo with a single core team just won't scale.

Where can I see a discussion of the Heat integration tests with Tempest
QA folks? If you give me some background on what efforts have been made
already and what is remaining to be reviewed/merged/worked on, then I
can try to get some resources dedicated to helping here.

I made a list at one point:


I'm not sure how complete it is, because a lot of those patches came from folks who were not Heat core members, and in some cases not even closely engaged in Heat development.

That wiki page was reviewed by the TC, although I was unable to make it to the meeting:


I would greatly prefer just having a single source of integration
testing in OpenStack, versus going back to the bad ol' days of everybody
under the sun rewriting their own.

I'd actually prefer some sort of plug-in system, where individual projects could supply tests and Tempest could load and run the appropriate ones for any particular test job in the correct environment. The Rally folks tell me that they support something like this for performance benchmarking... I'd really love it if the Rally and QA teams could work together to build something that benefits from *both* all the hard work the QA team has putting into making Tempest environments robust and some of the cool ideas in Rally.

If we continue to put all the integration tests in Tempest, then that means that every project will continue to gate against every other project, and that scales as O(n^2) in the number of projects. A plugin system would mean that e.g. Zaqar and Glance, say, would never gate against each other.

TBH I'd also be comfortable with e.g. Nova not gating against Heat. Gating against all dependent projects inevitably means the smaller, less mature projects will be in a position to accidentally cause chaos in the largest, most mature projects like Nova. I actually think it's going to be extremely rare that a change in Nova would break Heat, and on balance I think it's probably safer to just deal with that when it happens than to put Heat in a position where any problem with our tests can potentially grind Nova development to a halt.

That said, I absolutely want all of the python-*client projects to gate against Heat (and Horizon). So we would definitely be missing out if we can't find _some_ way to expose these integration tests outside of Heat.

Note that I'm not talking about functional testing here, just the
integration testing...

Yeah, you make a good point, and I don't think the distinction was actually clear (at least to me) until we discussed it in this thread. I probably shouldn't have replied glibly at 5.15pm on a Friday just before heading out of town for a week.


OpenStack-dev mailing list

Reply via email to