On 08/26/2014 10:14 AM, Zane Bitter wrote:
Steve Baker has started the process of moving Heat tests out of the
Tempest repository and into the Heat repository, and we're looking for
some guidance on how they should be packaged in a consistent way.
Apparently there are a few projects already packaging functional tests
in the package <projectname>.tests.functional (alongside
<projectname>.tests.unit for the unit tests).
That strikes me as odd in our context, because while the unit tests
run against the code in the package in which they are embedded, the
functional tests run against some entirely different code - whatever
OpenStack cloud you give it the auth URL and credentials for. So these
tests run from the outside, just like their ancestors in Tempest do.
There's all kinds of potential confusion here for users and packagers.
None of it is fatal and all of it can be worked around, but if we
refrain from doing the thing that makes zero conceptual sense then
there will be no problem to work around :)
Thanks, Zane. The point of moving functional tests to projects is to be
able to run more of them
in gate jobs for those projects, and allow tempest to survive being
stretched-to-breaking horizontally as we scale to more projects. At the
same time, there are benefits to the
tempest-as-all-in-one-functional-and-integration-suite that we should
try not to lose:
1. Strong integration testing without thinking too hard about the actual
dependencies
2. Protection from mistaken or unwise api changes (tempest two-step
required)
3. Exportability as a complete blackbox functional test suite that can
be used by Rally, RefStack, deployment validation, etc.
I think (1) may be the most challenging because tests that are moved out
of tempest might be testing some integration that is not being covered
by a scenario. We will need to make sure that tempest actually has a
complete enough set of tests to validate integration. Even if this is
all implemented in a way where tempest can see in-project tests as
"plugins", there will still not be time to run them all as part of
tempest on every commit to every project, so a selection will have to be
made.
(2) is quite difficult. In Atlanta we talked about taking a copy of
functional tests into tempest for stable apis. I don't know how workable
that is but don't see any other real options except vigilance in reviews
of patches that change functional tests.
(3) is what Zane was addressing. The in-project functional tests need to
be written in a way that they can, at least in some configuration, run
against a real cloud.
I suspect from reading the previous thread about "In-tree functional
test vision" that we may actually be dealing with three categories of
test here rather than two:
* Unit tests that run against the package they are embedded in
* Functional tests that run against the package they are embedded in
* Integration tests that run against a specified cloud
i.e. the tests we are now trying to add to Heat might be qualitatively
different from the <projectname>.tests.functional suites that already
exist in a few projects. Perhaps someone from Neutron and/or Swift can
confirm?
That seems right, except that I would call the third "functional tests"
and not "integration tests", because the purpose is not really
integration but deep testing of a particular service. Tempest would
continue to focus on integration testing. Is there some controversy
about that?
The second category could include whitebox tests.
I don't know about swift, but in neutron the intent was to have these
tests be configurable to run against a real cloud, or not. Maru Newby
would have details.
I'd like to propose that tests of the third type get their own
top-level package with a name of the form
<projectname>-integrationtests (second choice: <projectname>-tempest
on the principle that they're essentially plugins for Tempest). How
would people feel about standardising that across OpenStack?
+1 But I would not call it "integrationtests" for the reason given above.
-David
thanks,
Zane.
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
