Re: [openstack-dev] [qa][all][Heat] Packaging of functional tests

2014-09-09 Thread Zane Bitter

On 04/09/14 10:45, Jay Pipes wrote:

On 08/29/2014 05:15 PM, Zane Bitter wrote:

On 29/08/14 14:27, Jay Pipes wrote:

On 08/26/2014 10:14 AM, Zane Bitter wrote:

Steve Baker has started the process of moving Heat tests out of the
Tempest repository and into the Heat repository, and we're looking for
some guidance on how they should be packaged in a consistent way.
Apparently there are a few projects already packaging functional tests
in the package projectname.tests.functional (alongside
projectname.tests.unit for the unit tests).

That strikes me as odd in our context, because while the unit tests run
against the code in the package in which they are embedded, the
functional tests run against some entirely different code - whatever
OpenStack cloud you give it the auth URL and credentials for. So these
tests run from the outside, just like their ancestors in Tempest do.

There's all kinds of potential confusion here for users and packagers.
None of it is fatal and all of it can be worked around, but if we
refrain from doing the thing that makes zero conceptual sense then
there
will be no problem to work around :)

I suspect from reading the previous thread about In-tree functional
test vision that we may actually be dealing with three categories of
test here rather than two:

* Unit tests that run against the package they are embedded in
* Functional tests that run against the package they are embedded in
* Integration tests that run against a specified cloud

i.e. the tests we are now trying to add to Heat might be qualitatively
different from the projectname.tests.functional suites that already
exist in a few projects. Perhaps someone from Neutron and/or Swift can
confirm?

I'd like to propose that tests of the third type get their own
top-level
package with a name of the form projectname-integrationtests (second
choice: projectname-tempest on the principle that they're essentially
plugins for Tempest). How would people feel about standardising that
across OpenStack?


By its nature, Heat is one of the only projects that would have
integration tests of this nature. For Nova, there are some functional
tests in nova/tests/integrated/ (yeah, badly named, I know) that are
tests of the REST API endpoints and running service daemons (the things
that are RPC endpoints), with a bunch of stuff faked out (like RPC
comms, image services, authentication and the hypervisor layer itself).
So, the integrated tests in Nova are really not testing integration
with other projects, but rather integration of the subsystems and
processes inside Nova.

I'd support a policy that true integration tests -- tests that test the
interaction between multiple real OpenStack service endpoints -- be left
entirely to Tempest. Functional tests that test interaction between
internal daemons and processes to a project should go into
/$project/tests/functional/.

For Heat, I believe tests that rely on faked-out other OpenStack
services but stress the interaction between internal Heat
daemons/processes should be in /heat/tests/functional/ and any tests the
rely on working, real OpenStack service endpoints should be in Tempest.


Well, the problem with that is that last time I checked there was
exactly one Heat scenario test in Tempest because tempest-core doesn't
have the bandwidth to merge all (any?) of the other ones folks submitted.

So we're moving them to openstack/heat for the pure practical reason
that it's the only way to get test coverage at all, rather than concerns
about overloading the gate or theories about the best venue for
cross-project integration testing.


Hmm, speaking of passive aggressivity...


That's probably a fair criticism, in light of what Matt said about the 
failures of communication on both sides. I think a formal liaison 
program will be an enormous help here. However, it won't change the fact 
that keeping the tests for every project in a single repo with a single 
core team just won't scale.



Where can I see a discussion of the Heat integration tests with Tempest
QA folks? If you give me some background on what efforts have been made
already and what is remaining to be reviewed/merged/worked on, then I
can try to get some resources dedicated to helping here.


I made a list at one point:

https://wiki.openstack.org/w/index.php?title=Governance/TechnicalCommittee/Heat_Gap_Coverageoldid=58358#Improved_functional_testing_with_Tempest

I'm not sure how complete it is, because a lot of those patches came 
from folks who were not Heat core members, and in some cases not even 
closely engaged in Heat development.


That wiki page was reviewed by the TC, although I was unable to make it 
to the meeting:


http://eavesdrop.openstack.org/meetings/tc/2014/tc.2014-07-29-20.02.html


I would greatly prefer just having a single source of integration
testing in OpenStack, versus going back to the bad ol' days of everybody
under the sun rewriting their own.


I'd actually prefer some sort of plug-in system, where 

Re: [openstack-dev] [qa][all][Heat] Packaging of functional tests

2014-09-07 Thread Steve Baker
On 06/09/14 04:10, Matthew Treinish wrote:
 On Fri, Sep 05, 2014 at 09:42:17AM +1200, Steve Baker wrote:
 On 05/09/14 04:51, Matthew Treinish wrote:
 On Thu, Sep 04, 2014 at 04:32:53PM +0100, Steven Hardy wrote:
 On Thu, Sep 04, 2014 at 10:45:59AM -0400, Jay Pipes wrote:
 On 08/29/2014 05:15 PM, Zane Bitter wrote:
 On 29/08/14 14:27, Jay Pipes wrote:
 On 08/26/2014 10:14 AM, Zane Bitter wrote:
 Steve Baker has started the process of moving Heat tests out of the
 Tempest repository and into the Heat repository, and we're looking for
 some guidance on how they should be packaged in a consistent way.
 Apparently there are a few projects already packaging functional tests
 in the package projectname.tests.functional (alongside
 projectname.tests.unit for the unit tests).

 That strikes me as odd in our context, because while the unit tests run
 against the code in the package in which they are embedded, the
 functional tests run against some entirely different code - whatever
 OpenStack cloud you give it the auth URL and credentials for. So these
 tests run from the outside, just like their ancestors in Tempest do.

 There's all kinds of potential confusion here for users and packagers.
 None of it is fatal and all of it can be worked around, but if we
 refrain from doing the thing that makes zero conceptual sense then 
 there
 will be no problem to work around :)

 I suspect from reading the previous thread about In-tree functional
 test vision that we may actually be dealing with three categories of
 test here rather than two:

 * Unit tests that run against the package they are embedded in
 * Functional tests that run against the package they are embedded in
 * Integration tests that run against a specified cloud

 i.e. the tests we are now trying to add to Heat might be qualitatively
 different from the projectname.tests.functional suites that already
 exist in a few projects. Perhaps someone from Neutron and/or Swift can
 confirm?

 I'd like to propose that tests of the third type get their own 
 top-level
 package with a name of the form projectname-integrationtests (second
 choice: projectname-tempest on the principle that they're essentially
 plugins for Tempest). How would people feel about standardising that
 across OpenStack?
 By its nature, Heat is one of the only projects that would have
 integration tests of this nature. For Nova, there are some functional
 tests in nova/tests/integrated/ (yeah, badly named, I know) that are
 tests of the REST API endpoints and running service daemons (the things
 that are RPC endpoints), with a bunch of stuff faked out (like RPC
 comms, image services, authentication and the hypervisor layer itself).
 So, the integrated tests in Nova are really not testing integration
 with other projects, but rather integration of the subsystems and
 processes inside Nova.

 I'd support a policy that true integration tests -- tests that test the
 interaction between multiple real OpenStack service endpoints -- be left
 entirely to Tempest. Functional tests that test interaction between
 internal daemons and processes to a project should go into
 /$project/tests/functional/.

 For Heat, I believe tests that rely on faked-out other OpenStack
 services but stress the interaction between internal Heat
 daemons/processes should be in /heat/tests/functional/ and any tests the
 rely on working, real OpenStack service endpoints should be in Tempest.
 Well, the problem with that is that last time I checked there was
 exactly one Heat scenario test in Tempest because tempest-core doesn't
 have the bandwidth to merge all (any?) of the other ones folks submitted.

 So we're moving them to openstack/heat for the pure practical reason
 that it's the only way to get test coverage at all, rather than concerns
 about overloading the gate or theories about the best venue for
 cross-project integration testing.
 Hmm, speaking of passive aggressivity...

 Where can I see a discussion of the Heat integration tests with Tempest QA
 folks? If you give me some background on what efforts have been made 
 already
 and what is remaining to be reviewed/merged/worked on, then I can try to 
 get
 some resources dedicated to helping here.
 We recieved some fairly strong criticism from sdague[1] earlier this year,
 at which point we were  already actively working on improving test coverage
 by writing new tests for tempest.

 Since then, several folks, myself included, commited very significant
 amounts of additional effort to writing more tests for tempest, with some
 success.

 Ultimately the review latency and overhead involved in constantly rebasing
 changes between infrequent reviews has resulted in slow progress and
 significant frustration for those attempting to contribute new test cases.

 It's been clear for a while that tempest-core have significant bandwidth
 issues, as well as not necessarily always having the specific domain
 expertise to thoroughly review some tests related to project-specific

Re: [openstack-dev] [qa][all][Heat] Packaging of functional tests

2014-09-05 Thread Matthew Treinish
On Fri, Sep 05, 2014 at 09:42:17AM +1200, Steve Baker wrote:
 On 05/09/14 04:51, Matthew Treinish wrote:
  On Thu, Sep 04, 2014 at 04:32:53PM +0100, Steven Hardy wrote:
  On Thu, Sep 04, 2014 at 10:45:59AM -0400, Jay Pipes wrote:
  On 08/29/2014 05:15 PM, Zane Bitter wrote:
  On 29/08/14 14:27, Jay Pipes wrote:
  On 08/26/2014 10:14 AM, Zane Bitter wrote:
  Steve Baker has started the process of moving Heat tests out of the
  Tempest repository and into the Heat repository, and we're looking for
  some guidance on how they should be packaged in a consistent way.
  Apparently there are a few projects already packaging functional tests
  in the package projectname.tests.functional (alongside
  projectname.tests.unit for the unit tests).
 
  That strikes me as odd in our context, because while the unit tests run
  against the code in the package in which they are embedded, the
  functional tests run against some entirely different code - whatever
  OpenStack cloud you give it the auth URL and credentials for. So these
  tests run from the outside, just like their ancestors in Tempest do.
 
  There's all kinds of potential confusion here for users and packagers.
  None of it is fatal and all of it can be worked around, but if we
  refrain from doing the thing that makes zero conceptual sense then 
  there
  will be no problem to work around :)
 
  I suspect from reading the previous thread about In-tree functional
  test vision that we may actually be dealing with three categories of
  test here rather than two:
 
  * Unit tests that run against the package they are embedded in
  * Functional tests that run against the package they are embedded in
  * Integration tests that run against a specified cloud
 
  i.e. the tests we are now trying to add to Heat might be qualitatively
  different from the projectname.tests.functional suites that already
  exist in a few projects. Perhaps someone from Neutron and/or Swift can
  confirm?
 
  I'd like to propose that tests of the third type get their own 
  top-level
  package with a name of the form projectname-integrationtests (second
  choice: projectname-tempest on the principle that they're essentially
  plugins for Tempest). How would people feel about standardising that
  across OpenStack?
  By its nature, Heat is one of the only projects that would have
  integration tests of this nature. For Nova, there are some functional
  tests in nova/tests/integrated/ (yeah, badly named, I know) that are
  tests of the REST API endpoints and running service daemons (the things
  that are RPC endpoints), with a bunch of stuff faked out (like RPC
  comms, image services, authentication and the hypervisor layer itself).
  So, the integrated tests in Nova are really not testing integration
  with other projects, but rather integration of the subsystems and
  processes inside Nova.
 
  I'd support a policy that true integration tests -- tests that test the
  interaction between multiple real OpenStack service endpoints -- be left
  entirely to Tempest. Functional tests that test interaction between
  internal daemons and processes to a project should go into
  /$project/tests/functional/.
 
  For Heat, I believe tests that rely on faked-out other OpenStack
  services but stress the interaction between internal Heat
  daemons/processes should be in /heat/tests/functional/ and any tests the
  rely on working, real OpenStack service endpoints should be in Tempest.
  Well, the problem with that is that last time I checked there was
  exactly one Heat scenario test in Tempest because tempest-core doesn't
  have the bandwidth to merge all (any?) of the other ones folks submitted.
 
  So we're moving them to openstack/heat for the pure practical reason
  that it's the only way to get test coverage at all, rather than concerns
  about overloading the gate or theories about the best venue for
  cross-project integration testing.
  Hmm, speaking of passive aggressivity...
 
  Where can I see a discussion of the Heat integration tests with Tempest QA
  folks? If you give me some background on what efforts have been made 
  already
  and what is remaining to be reviewed/merged/worked on, then I can try to 
  get
  some resources dedicated to helping here.
  We recieved some fairly strong criticism from sdague[1] earlier this year,
  at which point we were  already actively working on improving test coverage
  by writing new tests for tempest.
 
  Since then, several folks, myself included, commited very significant
  amounts of additional effort to writing more tests for tempest, with some
  success.
 
  Ultimately the review latency and overhead involved in constantly rebasing
  changes between infrequent reviews has resulted in slow progress and
  significant frustration for those attempting to contribute new test cases.
 
  It's been clear for a while that tempest-core have significant bandwidth
  issues, as well as not necessarily always having the specific domain
  expertise to 

Re: [openstack-dev] [qa][all][Heat] Packaging of functional tests

2014-09-05 Thread David Kranz

On 09/05/2014 12:10 PM, Matthew Treinish wrote:

On Fri, Sep 05, 2014 at 09:42:17AM +1200, Steve Baker wrote:

On 05/09/14 04:51, Matthew Treinish wrote:

On Thu, Sep 04, 2014 at 04:32:53PM +0100, Steven Hardy wrote:

On Thu, Sep 04, 2014 at 10:45:59AM -0400, Jay Pipes wrote:

On 08/29/2014 05:15 PM, Zane Bitter wrote:

On 29/08/14 14:27, Jay Pipes wrote:

On 08/26/2014 10:14 AM, Zane Bitter wrote:

Steve Baker has started the process of moving Heat tests out of the
Tempest repository and into the Heat repository, and we're looking for
some guidance on how they should be packaged in a consistent way.
Apparently there are a few projects already packaging functional tests
in the package projectname.tests.functional (alongside
projectname.tests.unit for the unit tests).

That strikes me as odd in our context, because while the unit tests run
against the code in the package in which they are embedded, the
functional tests run against some entirely different code - whatever
OpenStack cloud you give it the auth URL and credentials for. So these
tests run from the outside, just like their ancestors in Tempest do.

There's all kinds of potential confusion here for users and packagers.
None of it is fatal and all of it can be worked around, but if we
refrain from doing the thing that makes zero conceptual sense then there
will be no problem to work around :)

I suspect from reading the previous thread about In-tree functional
test vision that we may actually be dealing with three categories of
test here rather than two:

* Unit tests that run against the package they are embedded in
* Functional tests that run against the package they are embedded in
* Integration tests that run against a specified cloud

i.e. the tests we are now trying to add to Heat might be qualitatively
different from the projectname.tests.functional suites that already
exist in a few projects. Perhaps someone from Neutron and/or Swift can
confirm?

I'd like to propose that tests of the third type get their own top-level
package with a name of the form projectname-integrationtests (second
choice: projectname-tempest on the principle that they're essentially
plugins for Tempest). How would people feel about standardising that
across OpenStack?

By its nature, Heat is one of the only projects that would have
integration tests of this nature. For Nova, there are some functional
tests in nova/tests/integrated/ (yeah, badly named, I know) that are
tests of the REST API endpoints and running service daemons (the things
that are RPC endpoints), with a bunch of stuff faked out (like RPC
comms, image services, authentication and the hypervisor layer itself).
So, the integrated tests in Nova are really not testing integration
with other projects, but rather integration of the subsystems and
processes inside Nova.

I'd support a policy that true integration tests -- tests that test the
interaction between multiple real OpenStack service endpoints -- be left
entirely to Tempest. Functional tests that test interaction between
internal daemons and processes to a project should go into
/$project/tests/functional/.

For Heat, I believe tests that rely on faked-out other OpenStack
services but stress the interaction between internal Heat
daemons/processes should be in /heat/tests/functional/ and any tests the
rely on working, real OpenStack service endpoints should be in Tempest.

Well, the problem with that is that last time I checked there was
exactly one Heat scenario test in Tempest because tempest-core doesn't
have the bandwidth to merge all (any?) of the other ones folks submitted.

So we're moving them to openstack/heat for the pure practical reason
that it's the only way to get test coverage at all, rather than concerns
about overloading the gate or theories about the best venue for
cross-project integration testing.

Hmm, speaking of passive aggressivity...

Where can I see a discussion of the Heat integration tests with Tempest QA
folks? If you give me some background on what efforts have been made already
and what is remaining to be reviewed/merged/worked on, then I can try to get
some resources dedicated to helping here.

We recieved some fairly strong criticism from sdague[1] earlier this year,
at which point we were  already actively working on improving test coverage
by writing new tests for tempest.

Since then, several folks, myself included, commited very significant
amounts of additional effort to writing more tests for tempest, with some
success.

Ultimately the review latency and overhead involved in constantly rebasing
changes between infrequent reviews has resulted in slow progress and
significant frustration for those attempting to contribute new test cases.

It's been clear for a while that tempest-core have significant bandwidth
issues, as well as not necessarily always having the specific domain
expertise to thoroughly review some tests related to project-specific
behavior or functionality.

So I view this as actually a breakdown 

Re: [openstack-dev] [qa][all][Heat] Packaging of functional tests

2014-09-04 Thread Jay Pipes

On 08/29/2014 05:15 PM, Zane Bitter wrote:

On 29/08/14 14:27, Jay Pipes wrote:

On 08/26/2014 10:14 AM, Zane Bitter wrote:

Steve Baker has started the process of moving Heat tests out of the
Tempest repository and into the Heat repository, and we're looking for
some guidance on how they should be packaged in a consistent way.
Apparently there are a few projects already packaging functional tests
in the package projectname.tests.functional (alongside
projectname.tests.unit for the unit tests).

That strikes me as odd in our context, because while the unit tests run
against the code in the package in which they are embedded, the
functional tests run against some entirely different code - whatever
OpenStack cloud you give it the auth URL and credentials for. So these
tests run from the outside, just like their ancestors in Tempest do.

There's all kinds of potential confusion here for users and packagers.
None of it is fatal and all of it can be worked around, but if we
refrain from doing the thing that makes zero conceptual sense then there
will be no problem to work around :)

I suspect from reading the previous thread about In-tree functional
test vision that we may actually be dealing with three categories of
test here rather than two:

* Unit tests that run against the package they are embedded in
* Functional tests that run against the package they are embedded in
* Integration tests that run against a specified cloud

i.e. the tests we are now trying to add to Heat might be qualitatively
different from the projectname.tests.functional suites that already
exist in a few projects. Perhaps someone from Neutron and/or Swift can
confirm?

I'd like to propose that tests of the third type get their own top-level
package with a name of the form projectname-integrationtests (second
choice: projectname-tempest on the principle that they're essentially
plugins for Tempest). How would people feel about standardising that
across OpenStack?


By its nature, Heat is one of the only projects that would have
integration tests of this nature. For Nova, there are some functional
tests in nova/tests/integrated/ (yeah, badly named, I know) that are
tests of the REST API endpoints and running service daemons (the things
that are RPC endpoints), with a bunch of stuff faked out (like RPC
comms, image services, authentication and the hypervisor layer itself).
So, the integrated tests in Nova are really not testing integration
with other projects, but rather integration of the subsystems and
processes inside Nova.

I'd support a policy that true integration tests -- tests that test the
interaction between multiple real OpenStack service endpoints -- be left
entirely to Tempest. Functional tests that test interaction between
internal daemons and processes to a project should go into
/$project/tests/functional/.

For Heat, I believe tests that rely on faked-out other OpenStack
services but stress the interaction between internal Heat
daemons/processes should be in /heat/tests/functional/ and any tests the
rely on working, real OpenStack service endpoints should be in Tempest.


Well, the problem with that is that last time I checked there was
exactly one Heat scenario test in Tempest because tempest-core doesn't
have the bandwidth to merge all (any?) of the other ones folks submitted.

So we're moving them to openstack/heat for the pure practical reason
that it's the only way to get test coverage at all, rather than concerns
about overloading the gate or theories about the best venue for
cross-project integration testing.


Hmm, speaking of passive aggressivity...

Where can I see a discussion of the Heat integration tests with Tempest 
QA folks? If you give me some background on what efforts have been made 
already and what is remaining to be reviewed/merged/worked on, then I 
can try to get some resources dedicated to helping here.


I would greatly prefer just having a single source of integration 
testing in OpenStack, versus going back to the bad ol' days of everybody 
under the sun rewriting their own.


Note that I'm not talking about functional testing here, just the 
integration testing...


Best,
-jay

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [qa][all][Heat] Packaging of functional tests

2014-09-04 Thread Steven Hardy
On Thu, Sep 04, 2014 at 10:45:59AM -0400, Jay Pipes wrote:
 On 08/29/2014 05:15 PM, Zane Bitter wrote:
 On 29/08/14 14:27, Jay Pipes wrote:
 On 08/26/2014 10:14 AM, Zane Bitter wrote:
 Steve Baker has started the process of moving Heat tests out of the
 Tempest repository and into the Heat repository, and we're looking for
 some guidance on how they should be packaged in a consistent way.
 Apparently there are a few projects already packaging functional tests
 in the package projectname.tests.functional (alongside
 projectname.tests.unit for the unit tests).
 
 That strikes me as odd in our context, because while the unit tests run
 against the code in the package in which they are embedded, the
 functional tests run against some entirely different code - whatever
 OpenStack cloud you give it the auth URL and credentials for. So these
 tests run from the outside, just like their ancestors in Tempest do.
 
 There's all kinds of potential confusion here for users and packagers.
 None of it is fatal and all of it can be worked around, but if we
 refrain from doing the thing that makes zero conceptual sense then there
 will be no problem to work around :)
 
 I suspect from reading the previous thread about In-tree functional
 test vision that we may actually be dealing with three categories of
 test here rather than two:
 
 * Unit tests that run against the package they are embedded in
 * Functional tests that run against the package they are embedded in
 * Integration tests that run against a specified cloud
 
 i.e. the tests we are now trying to add to Heat might be qualitatively
 different from the projectname.tests.functional suites that already
 exist in a few projects. Perhaps someone from Neutron and/or Swift can
 confirm?
 
 I'd like to propose that tests of the third type get their own top-level
 package with a name of the form projectname-integrationtests (second
 choice: projectname-tempest on the principle that they're essentially
 plugins for Tempest). How would people feel about standardising that
 across OpenStack?
 
 By its nature, Heat is one of the only projects that would have
 integration tests of this nature. For Nova, there are some functional
 tests in nova/tests/integrated/ (yeah, badly named, I know) that are
 tests of the REST API endpoints and running service daemons (the things
 that are RPC endpoints), with a bunch of stuff faked out (like RPC
 comms, image services, authentication and the hypervisor layer itself).
 So, the integrated tests in Nova are really not testing integration
 with other projects, but rather integration of the subsystems and
 processes inside Nova.
 
 I'd support a policy that true integration tests -- tests that test the
 interaction between multiple real OpenStack service endpoints -- be left
 entirely to Tempest. Functional tests that test interaction between
 internal daemons and processes to a project should go into
 /$project/tests/functional/.
 
 For Heat, I believe tests that rely on faked-out other OpenStack
 services but stress the interaction between internal Heat
 daemons/processes should be in /heat/tests/functional/ and any tests the
 rely on working, real OpenStack service endpoints should be in Tempest.
 
 Well, the problem with that is that last time I checked there was
 exactly one Heat scenario test in Tempest because tempest-core doesn't
 have the bandwidth to merge all (any?) of the other ones folks submitted.
 
 So we're moving them to openstack/heat for the pure practical reason
 that it's the only way to get test coverage at all, rather than concerns
 about overloading the gate or theories about the best venue for
 cross-project integration testing.
 
 Hmm, speaking of passive aggressivity...
 
 Where can I see a discussion of the Heat integration tests with Tempest QA
 folks? If you give me some background on what efforts have been made already
 and what is remaining to be reviewed/merged/worked on, then I can try to get
 some resources dedicated to helping here.

We recieved some fairly strong criticism from sdague[1] earlier this year,
at which point we were  already actively working on improving test coverage
by writing new tests for tempest.

Since then, several folks, myself included, commited very significant
amounts of additional effort to writing more tests for tempest, with some
success.

Ultimately the review latency and overhead involved in constantly rebasing
changes between infrequent reviews has resulted in slow progress and
significant frustration for those attempting to contribute new test cases.

It's been clear for a while that tempest-core have significant bandwidth
issues, as well as not necessarily always having the specific domain
expertise to thoroughly review some tests related to project-specific
behavior or functionality.

So it was with some relief that we saw the proposal[2] to move the burden
for reviewing project test-cases to the project teams, who will presumably
be more motivated to do the reviews, 

Re: [openstack-dev] [qa][all][Heat] Packaging of functional tests

2014-09-04 Thread Jay Pipes

On 09/04/2014 11:32 AM, Steven Hardy wrote:

On Thu, Sep 04, 2014 at 10:45:59AM -0400, Jay Pipes wrote:

On 08/29/2014 05:15 PM, Zane Bitter wrote:

On 29/08/14 14:27, Jay Pipes wrote:

On 08/26/2014 10:14 AM, Zane Bitter wrote:

Steve Baker has started the process of moving Heat tests out of the
Tempest repository and into the Heat repository, and we're looking for
some guidance on how they should be packaged in a consistent way.
Apparently there are a few projects already packaging functional tests
in the package projectname.tests.functional (alongside
projectname.tests.unit for the unit tests).

That strikes me as odd in our context, because while the unit tests run
against the code in the package in which they are embedded, the
functional tests run against some entirely different code - whatever
OpenStack cloud you give it the auth URL and credentials for. So these
tests run from the outside, just like their ancestors in Tempest do.

There's all kinds of potential confusion here for users and packagers.
None of it is fatal and all of it can be worked around, but if we
refrain from doing the thing that makes zero conceptual sense then there
will be no problem to work around :)

I suspect from reading the previous thread about In-tree functional
test vision that we may actually be dealing with three categories of
test here rather than two:

* Unit tests that run against the package they are embedded in
* Functional tests that run against the package they are embedded in
* Integration tests that run against a specified cloud

i.e. the tests we are now trying to add to Heat might be qualitatively
different from the projectname.tests.functional suites that already
exist in a few projects. Perhaps someone from Neutron and/or Swift can
confirm?

I'd like to propose that tests of the third type get their own top-level
package with a name of the form projectname-integrationtests (second
choice: projectname-tempest on the principle that they're essentially
plugins for Tempest). How would people feel about standardising that
across OpenStack?


By its nature, Heat is one of the only projects that would have
integration tests of this nature. For Nova, there are some functional
tests in nova/tests/integrated/ (yeah, badly named, I know) that are
tests of the REST API endpoints and running service daemons (the things
that are RPC endpoints), with a bunch of stuff faked out (like RPC
comms, image services, authentication and the hypervisor layer itself).
So, the integrated tests in Nova are really not testing integration
with other projects, but rather integration of the subsystems and
processes inside Nova.

I'd support a policy that true integration tests -- tests that test the
interaction between multiple real OpenStack service endpoints -- be left
entirely to Tempest. Functional tests that test interaction between
internal daemons and processes to a project should go into
/$project/tests/functional/.

For Heat, I believe tests that rely on faked-out other OpenStack
services but stress the interaction between internal Heat
daemons/processes should be in /heat/tests/functional/ and any tests the
rely on working, real OpenStack service endpoints should be in Tempest.


Well, the problem with that is that last time I checked there was
exactly one Heat scenario test in Tempest because tempest-core doesn't
have the bandwidth to merge all (any?) of the other ones folks submitted.

So we're moving them to openstack/heat for the pure practical reason
that it's the only way to get test coverage at all, rather than concerns
about overloading the gate or theories about the best venue for
cross-project integration testing.


Hmm, speaking of passive aggressivity...

Where can I see a discussion of the Heat integration tests with Tempest QA
folks? If you give me some background on what efforts have been made already
and what is remaining to be reviewed/merged/worked on, then I can try to get
some resources dedicated to helping here.


We recieved some fairly strong criticism from sdague[1] earlier this year,
at which point we were  already actively working on improving test coverage
by writing new tests for tempest.

Since then, several folks, myself included, commited very significant
amounts of additional effort to writing more tests for tempest, with some
success.

Ultimately the review latency and overhead involved in constantly rebasing
changes between infrequent reviews has resulted in slow progress and
significant frustration for those attempting to contribute new test cases.

It's been clear for a while that tempest-core have significant bandwidth
issues, as well as not necessarily always having the specific domain
expertise to thoroughly review some tests related to project-specific
behavior or functionality.

So it was with some relief that we saw the proposal[2] to move the burden
for reviewing project test-cases to the project teams, who will presumably
be more motivated to do the reviews, and have the knowledge 

Re: [openstack-dev] [qa][all][Heat] Packaging of functional tests

2014-09-04 Thread Matthew Treinish
On Thu, Sep 04, 2014 at 04:32:53PM +0100, Steven Hardy wrote:
 On Thu, Sep 04, 2014 at 10:45:59AM -0400, Jay Pipes wrote:
  On 08/29/2014 05:15 PM, Zane Bitter wrote:
  On 29/08/14 14:27, Jay Pipes wrote:
  On 08/26/2014 10:14 AM, Zane Bitter wrote:
  Steve Baker has started the process of moving Heat tests out of the
  Tempest repository and into the Heat repository, and we're looking for
  some guidance on how they should be packaged in a consistent way.
  Apparently there are a few projects already packaging functional tests
  in the package projectname.tests.functional (alongside
  projectname.tests.unit for the unit tests).
  
  That strikes me as odd in our context, because while the unit tests run
  against the code in the package in which they are embedded, the
  functional tests run against some entirely different code - whatever
  OpenStack cloud you give it the auth URL and credentials for. So these
  tests run from the outside, just like their ancestors in Tempest do.
  
  There's all kinds of potential confusion here for users and packagers.
  None of it is fatal and all of it can be worked around, but if we
  refrain from doing the thing that makes zero conceptual sense then there
  will be no problem to work around :)
  
  I suspect from reading the previous thread about In-tree functional
  test vision that we may actually be dealing with three categories of
  test here rather than two:
  
  * Unit tests that run against the package they are embedded in
  * Functional tests that run against the package they are embedded in
  * Integration tests that run against a specified cloud
  
  i.e. the tests we are now trying to add to Heat might be qualitatively
  different from the projectname.tests.functional suites that already
  exist in a few projects. Perhaps someone from Neutron and/or Swift can
  confirm?
  
  I'd like to propose that tests of the third type get their own top-level
  package with a name of the form projectname-integrationtests (second
  choice: projectname-tempest on the principle that they're essentially
  plugins for Tempest). How would people feel about standardising that
  across OpenStack?
  
  By its nature, Heat is one of the only projects that would have
  integration tests of this nature. For Nova, there are some functional
  tests in nova/tests/integrated/ (yeah, badly named, I know) that are
  tests of the REST API endpoints and running service daemons (the things
  that are RPC endpoints), with a bunch of stuff faked out (like RPC
  comms, image services, authentication and the hypervisor layer itself).
  So, the integrated tests in Nova are really not testing integration
  with other projects, but rather integration of the subsystems and
  processes inside Nova.
  
  I'd support a policy that true integration tests -- tests that test the
  interaction between multiple real OpenStack service endpoints -- be left
  entirely to Tempest. Functional tests that test interaction between
  internal daemons and processes to a project should go into
  /$project/tests/functional/.
  
  For Heat, I believe tests that rely on faked-out other OpenStack
  services but stress the interaction between internal Heat
  daemons/processes should be in /heat/tests/functional/ and any tests the
  rely on working, real OpenStack service endpoints should be in Tempest.
  
  Well, the problem with that is that last time I checked there was
  exactly one Heat scenario test in Tempest because tempest-core doesn't
  have the bandwidth to merge all (any?) of the other ones folks submitted.
  
  So we're moving them to openstack/heat for the pure practical reason
  that it's the only way to get test coverage at all, rather than concerns
  about overloading the gate or theories about the best venue for
  cross-project integration testing.
  
  Hmm, speaking of passive aggressivity...
  
  Where can I see a discussion of the Heat integration tests with Tempest QA
  folks? If you give me some background on what efforts have been made already
  and what is remaining to be reviewed/merged/worked on, then I can try to get
  some resources dedicated to helping here.
 
 We recieved some fairly strong criticism from sdague[1] earlier this year,
 at which point we were  already actively working on improving test coverage
 by writing new tests for tempest.
 
 Since then, several folks, myself included, commited very significant
 amounts of additional effort to writing more tests for tempest, with some
 success.
 
 Ultimately the review latency and overhead involved in constantly rebasing
 changes between infrequent reviews has resulted in slow progress and
 significant frustration for those attempting to contribute new test cases.
 
 It's been clear for a while that tempest-core have significant bandwidth
 issues, as well as not necessarily always having the specific domain
 expertise to thoroughly review some tests related to project-specific
 behavior or functionality.

So I view this as actually a 

Re: [openstack-dev] [qa][all][Heat] Packaging of functional tests

2014-09-04 Thread Steve Baker
On 05/09/14 04:51, Matthew Treinish wrote:
 On Thu, Sep 04, 2014 at 04:32:53PM +0100, Steven Hardy wrote:
 On Thu, Sep 04, 2014 at 10:45:59AM -0400, Jay Pipes wrote:
 On 08/29/2014 05:15 PM, Zane Bitter wrote:
 On 29/08/14 14:27, Jay Pipes wrote:
 On 08/26/2014 10:14 AM, Zane Bitter wrote:
 Steve Baker has started the process of moving Heat tests out of the
 Tempest repository and into the Heat repository, and we're looking for
 some guidance on how they should be packaged in a consistent way.
 Apparently there are a few projects already packaging functional tests
 in the package projectname.tests.functional (alongside
 projectname.tests.unit for the unit tests).

 That strikes me as odd in our context, because while the unit tests run
 against the code in the package in which they are embedded, the
 functional tests run against some entirely different code - whatever
 OpenStack cloud you give it the auth URL and credentials for. So these
 tests run from the outside, just like their ancestors in Tempest do.

 There's all kinds of potential confusion here for users and packagers.
 None of it is fatal and all of it can be worked around, but if we
 refrain from doing the thing that makes zero conceptual sense then there
 will be no problem to work around :)

 I suspect from reading the previous thread about In-tree functional
 test vision that we may actually be dealing with three categories of
 test here rather than two:

 * Unit tests that run against the package they are embedded in
 * Functional tests that run against the package they are embedded in
 * Integration tests that run against a specified cloud

 i.e. the tests we are now trying to add to Heat might be qualitatively
 different from the projectname.tests.functional suites that already
 exist in a few projects. Perhaps someone from Neutron and/or Swift can
 confirm?

 I'd like to propose that tests of the third type get their own top-level
 package with a name of the form projectname-integrationtests (second
 choice: projectname-tempest on the principle that they're essentially
 plugins for Tempest). How would people feel about standardising that
 across OpenStack?
 By its nature, Heat is one of the only projects that would have
 integration tests of this nature. For Nova, there are some functional
 tests in nova/tests/integrated/ (yeah, badly named, I know) that are
 tests of the REST API endpoints and running service daemons (the things
 that are RPC endpoints), with a bunch of stuff faked out (like RPC
 comms, image services, authentication and the hypervisor layer itself).
 So, the integrated tests in Nova are really not testing integration
 with other projects, but rather integration of the subsystems and
 processes inside Nova.

 I'd support a policy that true integration tests -- tests that test the
 interaction between multiple real OpenStack service endpoints -- be left
 entirely to Tempest. Functional tests that test interaction between
 internal daemons and processes to a project should go into
 /$project/tests/functional/.

 For Heat, I believe tests that rely on faked-out other OpenStack
 services but stress the interaction between internal Heat
 daemons/processes should be in /heat/tests/functional/ and any tests the
 rely on working, real OpenStack service endpoints should be in Tempest.
 Well, the problem with that is that last time I checked there was
 exactly one Heat scenario test in Tempest because tempest-core doesn't
 have the bandwidth to merge all (any?) of the other ones folks submitted.

 So we're moving them to openstack/heat for the pure practical reason
 that it's the only way to get test coverage at all, rather than concerns
 about overloading the gate or theories about the best venue for
 cross-project integration testing.
 Hmm, speaking of passive aggressivity...

 Where can I see a discussion of the Heat integration tests with Tempest QA
 folks? If you give me some background on what efforts have been made already
 and what is remaining to be reviewed/merged/worked on, then I can try to get
 some resources dedicated to helping here.
 We recieved some fairly strong criticism from sdague[1] earlier this year,
 at which point we were  already actively working on improving test coverage
 by writing new tests for tempest.

 Since then, several folks, myself included, commited very significant
 amounts of additional effort to writing more tests for tempest, with some
 success.

 Ultimately the review latency and overhead involved in constantly rebasing
 changes between infrequent reviews has resulted in slow progress and
 significant frustration for those attempting to contribute new test cases.

 It's been clear for a while that tempest-core have significant bandwidth
 issues, as well as not necessarily always having the specific domain
 expertise to thoroughly review some tests related to project-specific
 behavior or functionality.
 So I view this as actually a breakdown in cross-team communication, with both
 sides 

Re: [openstack-dev] [qa][all][Heat] Packaging of functional tests

2014-09-04 Thread Angus Salkeld
On Fri, Sep 5, 2014 at 2:40 AM, Jay Pipes jaypi...@gmail.com wrote:

 On 09/04/2014 11:32 AM, Steven Hardy wrote:

 On Thu, Sep 04, 2014 at 10:45:59AM -0400, Jay Pipes wrote:

 On 08/29/2014 05:15 PM, Zane Bitter wrote:

 On 29/08/14 14:27, Jay Pipes wrote:

 On 08/26/2014 10:14 AM, Zane Bitter wrote:

 Steve Baker has started the process of moving Heat tests out of the
 Tempest repository and into the Heat repository, and we're looking for
 some guidance on how they should be packaged in a consistent way.
 Apparently there are a few projects already packaging functional tests
 in the package projectname.tests.functional (alongside
 projectname.tests.unit for the unit tests).

 That strikes me as odd in our context, because while the unit tests
 run
 against the code in the package in which they are embedded, the
 functional tests run against some entirely different code - whatever
 OpenStack cloud you give it the auth URL and credentials for. So these
 tests run from the outside, just like their ancestors in Tempest do.

 There's all kinds of potential confusion here for users and packagers.
 None of it is fatal and all of it can be worked around, but if we
 refrain from doing the thing that makes zero conceptual sense then
 there
 will be no problem to work around :)

 I suspect from reading the previous thread about In-tree functional
 test vision that we may actually be dealing with three categories of
 test here rather than two:

 * Unit tests that run against the package they are embedded in
 * Functional tests that run against the package they are embedded in
 * Integration tests that run against a specified cloud

 i.e. the tests we are now trying to add to Heat might be qualitatively
 different from the projectname.tests.functional suites that already
 exist in a few projects. Perhaps someone from Neutron and/or Swift can
 confirm?

 I'd like to propose that tests of the third type get their own
 top-level
 package with a name of the form projectname-integrationtests (second
 choice: projectname-tempest on the principle that they're
 essentially
 plugins for Tempest). How would people feel about standardising that
 across OpenStack?


 By its nature, Heat is one of the only projects that would have
 integration tests of this nature. For Nova, there are some functional
 tests in nova/tests/integrated/ (yeah, badly named, I know) that are
 tests of the REST API endpoints and running service daemons (the things
 that are RPC endpoints), with a bunch of stuff faked out (like RPC
 comms, image services, authentication and the hypervisor layer itself).
 So, the integrated tests in Nova are really not testing integration
 with other projects, but rather integration of the subsystems and
 processes inside Nova.

 I'd support a policy that true integration tests -- tests that test the
 interaction between multiple real OpenStack service endpoints -- be
 left
 entirely to Tempest. Functional tests that test interaction between
 internal daemons and processes to a project should go into
 /$project/tests/functional/.

 For Heat, I believe tests that rely on faked-out other OpenStack
 services but stress the interaction between internal Heat
 daemons/processes should be in /heat/tests/functional/ and any tests
 the
 rely on working, real OpenStack service endpoints should be in Tempest.


 Well, the problem with that is that last time I checked there was
 exactly one Heat scenario test in Tempest because tempest-core doesn't
 have the bandwidth to merge all (any?) of the other ones folks
 submitted.

 So we're moving them to openstack/heat for the pure practical reason
 that it's the only way to get test coverage at all, rather than concerns
 about overloading the gate or theories about the best venue for
 cross-project integration testing.


 Hmm, speaking of passive aggressivity...

 Where can I see a discussion of the Heat integration tests with Tempest
 QA
 folks? If you give me some background on what efforts have been made
 already
 and what is remaining to be reviewed/merged/worked on, then I can try to
 get
 some resources dedicated to helping here.


 We recieved some fairly strong criticism from sdague[1] earlier this year,
 at which point we were  already actively working on improving test
 coverage
 by writing new tests for tempest.

 Since then, several folks, myself included, commited very significant
 amounts of additional effort to writing more tests for tempest, with some
 success.

 Ultimately the review latency and overhead involved in constantly rebasing
 changes between infrequent reviews has resulted in slow progress and
 significant frustration for those attempting to contribute new test cases.

 It's been clear for a while that tempest-core have significant bandwidth
 issues, as well as not necessarily always having the specific domain
 expertise to thoroughly review some tests related to project-specific
 behavior or functionality.

 So it was with some relief that we saw the 

Re: [openstack-dev] [qa][all][Heat] Packaging of functional tests

2014-08-29 Thread Jay Pipes

On 08/26/2014 10:14 AM, Zane Bitter wrote:

Steve Baker has started the process of moving Heat tests out of the
Tempest repository and into the Heat repository, and we're looking for
some guidance on how they should be packaged in a consistent way.
Apparently there are a few projects already packaging functional tests
in the package projectname.tests.functional (alongside
projectname.tests.unit for the unit tests).

That strikes me as odd in our context, because while the unit tests run
against the code in the package in which they are embedded, the
functional tests run against some entirely different code - whatever
OpenStack cloud you give it the auth URL and credentials for. So these
tests run from the outside, just like their ancestors in Tempest do.

There's all kinds of potential confusion here for users and packagers.
None of it is fatal and all of it can be worked around, but if we
refrain from doing the thing that makes zero conceptual sense then there
will be no problem to work around :)

I suspect from reading the previous thread about In-tree functional
test vision that we may actually be dealing with three categories of
test here rather than two:

* Unit tests that run against the package they are embedded in
* Functional tests that run against the package they are embedded in
* Integration tests that run against a specified cloud

i.e. the tests we are now trying to add to Heat might be qualitatively
different from the projectname.tests.functional suites that already
exist in a few projects. Perhaps someone from Neutron and/or Swift can
confirm?

I'd like to propose that tests of the third type get their own top-level
package with a name of the form projectname-integrationtests (second
choice: projectname-tempest on the principle that they're essentially
plugins for Tempest). How would people feel about standardising that
across OpenStack?


By its nature, Heat is one of the only projects that would have 
integration tests of this nature. For Nova, there are some functional 
tests in nova/tests/integrated/ (yeah, badly named, I know) that are 
tests of the REST API endpoints and running service daemons (the things 
that are RPC endpoints), with a bunch of stuff faked out (like RPC 
comms, image services, authentication and the hypervisor layer itself). 
So, the integrated tests in Nova are really not testing integration 
with other projects, but rather integration of the subsystems and 
processes inside Nova.


I'd support a policy that true integration tests -- tests that test the 
interaction between multiple real OpenStack service endpoints -- be left 
entirely to Tempest. Functional tests that test interaction between 
internal daemons and processes to a project should go into 
/$project/tests/functional/.


For Heat, I believe tests that rely on faked-out other OpenStack 
services but stress the interaction between internal Heat 
daemons/processes should be in /heat/tests/functional/ and any tests the 
rely on working, real OpenStack service endpoints should be in Tempest.


Best,
-jay

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [qa][all][Heat] Packaging of functional tests

2014-08-29 Thread Zane Bitter

On 29/08/14 14:27, Jay Pipes wrote:

On 08/26/2014 10:14 AM, Zane Bitter wrote:

Steve Baker has started the process of moving Heat tests out of the
Tempest repository and into the Heat repository, and we're looking for
some guidance on how they should be packaged in a consistent way.
Apparently there are a few projects already packaging functional tests
in the package projectname.tests.functional (alongside
projectname.tests.unit for the unit tests).

That strikes me as odd in our context, because while the unit tests run
against the code in the package in which they are embedded, the
functional tests run against some entirely different code - whatever
OpenStack cloud you give it the auth URL and credentials for. So these
tests run from the outside, just like their ancestors in Tempest do.

There's all kinds of potential confusion here for users and packagers.
None of it is fatal and all of it can be worked around, but if we
refrain from doing the thing that makes zero conceptual sense then there
will be no problem to work around :)

I suspect from reading the previous thread about In-tree functional
test vision that we may actually be dealing with three categories of
test here rather than two:

* Unit tests that run against the package they are embedded in
* Functional tests that run against the package they are embedded in
* Integration tests that run against a specified cloud

i.e. the tests we are now trying to add to Heat might be qualitatively
different from the projectname.tests.functional suites that already
exist in a few projects. Perhaps someone from Neutron and/or Swift can
confirm?

I'd like to propose that tests of the third type get their own top-level
package with a name of the form projectname-integrationtests (second
choice: projectname-tempest on the principle that they're essentially
plugins for Tempest). How would people feel about standardising that
across OpenStack?


By its nature, Heat is one of the only projects that would have
integration tests of this nature. For Nova, there are some functional
tests in nova/tests/integrated/ (yeah, badly named, I know) that are
tests of the REST API endpoints and running service daemons (the things
that are RPC endpoints), with a bunch of stuff faked out (like RPC
comms, image services, authentication and the hypervisor layer itself).
So, the integrated tests in Nova are really not testing integration
with other projects, but rather integration of the subsystems and
processes inside Nova.

I'd support a policy that true integration tests -- tests that test the
interaction between multiple real OpenStack service endpoints -- be left
entirely to Tempest. Functional tests that test interaction between
internal daemons and processes to a project should go into
/$project/tests/functional/.

For Heat, I believe tests that rely on faked-out other OpenStack
services but stress the interaction between internal Heat
daemons/processes should be in /heat/tests/functional/ and any tests the
rely on working, real OpenStack service endpoints should be in Tempest.


Well, the problem with that is that last time I checked there was 
exactly one Heat scenario test in Tempest because tempest-core doesn't 
have the bandwidth to merge all (any?) of the other ones folks submitted.


So we're moving them to openstack/heat for the pure practical reason 
that it's the only way to get test coverage at all, rather than concerns 
about overloading the gate or theories about the best venue for 
cross-project integration testing.


cheers,
Zane.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [qa][all][Heat] Packaging of functional tests

2014-08-27 Thread Zane Bitter

On 26/08/14 18:59, Clint Byrum wrote:

Excerpts from Steve Baker's message of 2014-08-26 14:25:46 -0700:

On 27/08/14 03:18, David Kranz wrote:

On 08/26/2014 10:14 AM, Zane Bitter wrote:

Steve Baker has started the process of moving Heat tests out of the
Tempest repository and into the Heat repository, and we're looking
for some guidance on how they should be packaged in a consistent way.
Apparently there are a few projects already packaging functional
tests in the package projectname.tests.functional (alongside
projectname.tests.unit for the unit tests).

That strikes me as odd in our context, because while the unit tests
run against the code in the package in which they are embedded, the
functional tests run against some entirely different code - whatever
OpenStack cloud you give it the auth URL and credentials for. So
these tests run from the outside, just like their ancestors in
Tempest do.

There's all kinds of potential confusion here for users and
packagers. None of it is fatal and all of it can be worked around,
but if we refrain from doing the thing that makes zero conceptual
sense then there will be no problem to work around :)

Thanks, Zane. The point of moving functional tests to projects is to
be able to run more of them
in gate jobs for those projects, and allow tempest to survive being
stretched-to-breaking horizontally as we scale to more projects. At
the same time, there are benefits to the
tempest-as-all-in-one-functional-and-integration-suite that we should
try not to lose:

1. Strong integration testing without thinking too hard about the
actual dependencies
2. Protection from mistaken or unwise api changes (tempest two-step
required)
3. Exportability as a complete blackbox functional test suite that can
be used by Rally, RefStack, deployment validation, etc.

I think (1) may be the most challenging because tests that are moved
out of tempest might be testing some integration that is not being
covered
by a scenario. We will need to make sure that tempest actually has a
complete enough set of tests to validate integration. Even if this is
all implemented in a way where tempest can see in-project tests as
plugins, there will still not be time to run them all as part of
tempest on every commit to every project, so a selection will have to
be made.

(2) is quite difficult. In Atlanta we talked about taking a copy of
functional tests into tempest for stable apis. I don't know how
workable that is but don't see any other real options except vigilance
in reviews of patches that change functional tests.

(3) is what Zane was addressing. The in-project functional tests need
to be written in a way that they can, at least in some configuration,
run against a real cloud.




I suspect from reading the previous thread about In-tree functional
test vision that we may actually be dealing with three categories of
test here rather than two:

* Unit tests that run against the package they are embedded in
* Functional tests that run against the package they are embedded in
* Integration tests that run against a specified cloud

i.e. the tests we are now trying to add to Heat might be
qualitatively different from the projectname.tests.functional
suites that already exist in a few projects. Perhaps someone from
Neutron and/or Swift can confirm?

That seems right, except that I would call the third functional
tests and not integration tests, because the purpose is not really
integration but deep testing of a particular service. Tempest would
continue to focus on integration testing. Is there some controversy
about that?
The second category could include whitebox tests.

I don't know about swift, but in neutron the intent was to have these
tests be configurable to run against a real cloud, or not. Maru Newby
would have details.


I'd like to propose that tests of the third type get their own
top-level package with a name of the form
projectname-integrationtests (second choice: projectname-tempest
on the principle that they're essentially plugins for Tempest). How
would people feel about standardising that across OpenStack?

+1 But I would not call it integrationtests for the reason given above.


Because all heat does is interact with other services, what we call
functional tests are actually integration tests. Sure, we could mock at
the REST API level, but integration coverage is what we need most. This


I'd call that faking, not mocking, but both could apply.


lets us verify things like:
- how heat handles races in other services leading to resources going
into ERROR


A fake that predictably fails (and thus tests failure handling) will
result in better coverage than a real service that only fails when that
real service is broken. What's frustrating is that _both_ are needed to
catch bugs.


Yeah, we discussed this in the project meeting yesterday[1] and 
basically came to the conclusion that we'll ultimately need both 
functional tests with fake services and integration tests with real 
services.



Re: [openstack-dev] [qa][all][Heat] Packaging of functional tests

2014-08-26 Thread David Kranz

On 08/26/2014 10:14 AM, Zane Bitter wrote:
Steve Baker has started the process of moving Heat tests out of the 
Tempest repository and into the Heat repository, and we're looking for 
some guidance on how they should be packaged in a consistent way. 
Apparently there are a few projects already packaging functional tests 
in the package projectname.tests.functional (alongside 
projectname.tests.unit for the unit tests).


That strikes me as odd in our context, because while the unit tests 
run against the code in the package in which they are embedded, the 
functional tests run against some entirely different code - whatever 
OpenStack cloud you give it the auth URL and credentials for. So these 
tests run from the outside, just like their ancestors in Tempest do.


There's all kinds of potential confusion here for users and packagers. 
None of it is fatal and all of it can be worked around, but if we 
refrain from doing the thing that makes zero conceptual sense then 
there will be no problem to work around :)
Thanks, Zane. The point of moving functional tests to projects is to be 
able to run more of them
in gate jobs for those projects, and allow tempest to survive being 
stretched-to-breaking horizontally as we scale to more projects. At the 
same time, there are benefits to the 
tempest-as-all-in-one-functional-and-integration-suite that we should 
try not to lose:


1. Strong integration testing without thinking too hard about the actual 
dependencies
2. Protection from mistaken or unwise api changes (tempest two-step 
required)
3. Exportability as a complete blackbox functional test suite that can 
be used by Rally, RefStack, deployment validation, etc.


I think (1) may be the most challenging because tests that are moved out 
of tempest might be testing some integration that is not being covered
by a scenario. We will need to make sure that tempest actually has a 
complete enough set of tests to validate integration. Even if this is 
all implemented in a way where tempest can see in-project tests as 
plugins, there will still not be time to run them all as part of 
tempest on every commit to every project, so a selection will have to be 
made.


(2) is quite difficult. In Atlanta we talked about taking a copy of 
functional tests into tempest for stable apis. I don't know how workable 
that is but don't see any other real options except vigilance in reviews 
of patches that change functional tests.


(3) is what Zane was addressing. The in-project functional tests need to 
be written in a way that they can, at least in some configuration, run 
against a real cloud.





I suspect from reading the previous thread about In-tree functional 
test vision that we may actually be dealing with three categories of 
test here rather than two:


* Unit tests that run against the package they are embedded in
* Functional tests that run against the package they are embedded in
* Integration tests that run against a specified cloud

i.e. the tests we are now trying to add to Heat might be qualitatively 
different from the projectname.tests.functional suites that already 
exist in a few projects. Perhaps someone from Neutron and/or Swift can 
confirm?
That seems right, except that I would call the third functional tests 
and not integration tests, because the purpose is not really 
integration but deep testing of a particular service. Tempest would 
continue to focus on integration testing. Is there some controversy 
about that?

The second category could include whitebox tests.

I don't know about swift, but in neutron the intent was to have these 
tests be configurable to run against a real cloud, or not. Maru Newby 
would have details.


I'd like to propose that tests of the third type get their own 
top-level package with a name of the form 
projectname-integrationtests (second choice: projectname-tempest 
on the principle that they're essentially plugins for Tempest). How 
would people feel about standardising that across OpenStack?

+1 But I would not call it integrationtests for the reason given above.

 -David


thanks,
Zane.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [qa][all][Heat] Packaging of functional tests

2014-08-26 Thread Steve Baker
On 27/08/14 03:18, David Kranz wrote:
 On 08/26/2014 10:14 AM, Zane Bitter wrote:
 Steve Baker has started the process of moving Heat tests out of the
 Tempest repository and into the Heat repository, and we're looking
 for some guidance on how they should be packaged in a consistent way.
 Apparently there are a few projects already packaging functional
 tests in the package projectname.tests.functional (alongside
 projectname.tests.unit for the unit tests).

 That strikes me as odd in our context, because while the unit tests
 run against the code in the package in which they are embedded, the
 functional tests run against some entirely different code - whatever
 OpenStack cloud you give it the auth URL and credentials for. So
 these tests run from the outside, just like their ancestors in
 Tempest do.

 There's all kinds of potential confusion here for users and
 packagers. None of it is fatal and all of it can be worked around,
 but if we refrain from doing the thing that makes zero conceptual
 sense then there will be no problem to work around :)
 Thanks, Zane. The point of moving functional tests to projects is to
 be able to run more of them
 in gate jobs for those projects, and allow tempest to survive being
 stretched-to-breaking horizontally as we scale to more projects. At
 the same time, there are benefits to the
 tempest-as-all-in-one-functional-and-integration-suite that we should
 try not to lose:

 1. Strong integration testing without thinking too hard about the
 actual dependencies
 2. Protection from mistaken or unwise api changes (tempest two-step
 required)
 3. Exportability as a complete blackbox functional test suite that can
 be used by Rally, RefStack, deployment validation, etc.

 I think (1) may be the most challenging because tests that are moved
 out of tempest might be testing some integration that is not being
 covered
 by a scenario. We will need to make sure that tempest actually has a
 complete enough set of tests to validate integration. Even if this is
 all implemented in a way where tempest can see in-project tests as
 plugins, there will still not be time to run them all as part of
 tempest on every commit to every project, so a selection will have to
 be made.

 (2) is quite difficult. In Atlanta we talked about taking a copy of
 functional tests into tempest for stable apis. I don't know how
 workable that is but don't see any other real options except vigilance
 in reviews of patches that change functional tests.

 (3) is what Zane was addressing. The in-project functional tests need
 to be written in a way that they can, at least in some configuration,
 run against a real cloud.



 I suspect from reading the previous thread about In-tree functional
 test vision that we may actually be dealing with three categories of
 test here rather than two:

 * Unit tests that run against the package they are embedded in
 * Functional tests that run against the package they are embedded in
 * Integration tests that run against a specified cloud

 i.e. the tests we are now trying to add to Heat might be
 qualitatively different from the projectname.tests.functional
 suites that already exist in a few projects. Perhaps someone from
 Neutron and/or Swift can confirm?
 That seems right, except that I would call the third functional
 tests and not integration tests, because the purpose is not really
 integration but deep testing of a particular service. Tempest would
 continue to focus on integration testing. Is there some controversy
 about that?
 The second category could include whitebox tests.

 I don't know about swift, but in neutron the intent was to have these
 tests be configurable to run against a real cloud, or not. Maru Newby
 would have details.

 I'd like to propose that tests of the third type get their own
 top-level package with a name of the form
 projectname-integrationtests (second choice: projectname-tempest
 on the principle that they're essentially plugins for Tempest). How
 would people feel about standardising that across OpenStack?
 +1 But I would not call it integrationtests for the reason given above.

Because all heat does is interact with other services, what we call
functional tests are actually integration tests. Sure, we could mock at
the REST API level, but integration coverage is what we need most. This
lets us verify things like:
- how heat handles races in other services leading to resources going
into ERROR
- connectivity and interaction between heat and agents on orchestrated
servers

So I'm +1 on Zane's suggestion for projectname-integrationtests for
the sort of scenario integration testing we want to do.

I would imagine the swift and neutron functional tests are much more
self-contained, possibly only interacting with keystone. The heat API
tests are very much in this category, which currently live in
tempest.api.orchestration. These can move into the heat tree using
whatever established convention once tempest-lib is consumable, and/or

Re: [openstack-dev] [qa][all][Heat] Packaging of functional tests

2014-08-26 Thread Clint Byrum
Excerpts from Steve Baker's message of 2014-08-26 14:25:46 -0700:
 On 27/08/14 03:18, David Kranz wrote:
  On 08/26/2014 10:14 AM, Zane Bitter wrote:
  Steve Baker has started the process of moving Heat tests out of the
  Tempest repository and into the Heat repository, and we're looking
  for some guidance on how they should be packaged in a consistent way.
  Apparently there are a few projects already packaging functional
  tests in the package projectname.tests.functional (alongside
  projectname.tests.unit for the unit tests).
 
  That strikes me as odd in our context, because while the unit tests
  run against the code in the package in which they are embedded, the
  functional tests run against some entirely different code - whatever
  OpenStack cloud you give it the auth URL and credentials for. So
  these tests run from the outside, just like their ancestors in
  Tempest do.
 
  There's all kinds of potential confusion here for users and
  packagers. None of it is fatal and all of it can be worked around,
  but if we refrain from doing the thing that makes zero conceptual
  sense then there will be no problem to work around :)
  Thanks, Zane. The point of moving functional tests to projects is to
  be able to run more of them
  in gate jobs for those projects, and allow tempest to survive being
  stretched-to-breaking horizontally as we scale to more projects. At
  the same time, there are benefits to the
  tempest-as-all-in-one-functional-and-integration-suite that we should
  try not to lose:
 
  1. Strong integration testing without thinking too hard about the
  actual dependencies
  2. Protection from mistaken or unwise api changes (tempest two-step
  required)
  3. Exportability as a complete blackbox functional test suite that can
  be used by Rally, RefStack, deployment validation, etc.
 
  I think (1) may be the most challenging because tests that are moved
  out of tempest might be testing some integration that is not being
  covered
  by a scenario. We will need to make sure that tempest actually has a
  complete enough set of tests to validate integration. Even if this is
  all implemented in a way where tempest can see in-project tests as
  plugins, there will still not be time to run them all as part of
  tempest on every commit to every project, so a selection will have to
  be made.
 
  (2) is quite difficult. In Atlanta we talked about taking a copy of
  functional tests into tempest for stable apis. I don't know how
  workable that is but don't see any other real options except vigilance
  in reviews of patches that change functional tests.
 
  (3) is what Zane was addressing. The in-project functional tests need
  to be written in a way that they can, at least in some configuration,
  run against a real cloud.
 
 
 
  I suspect from reading the previous thread about In-tree functional
  test vision that we may actually be dealing with three categories of
  test here rather than two:
 
  * Unit tests that run against the package they are embedded in
  * Functional tests that run against the package they are embedded in
  * Integration tests that run against a specified cloud
 
  i.e. the tests we are now trying to add to Heat might be
  qualitatively different from the projectname.tests.functional
  suites that already exist in a few projects. Perhaps someone from
  Neutron and/or Swift can confirm?
  That seems right, except that I would call the third functional
  tests and not integration tests, because the purpose is not really
  integration but deep testing of a particular service. Tempest would
  continue to focus on integration testing. Is there some controversy
  about that?
  The second category could include whitebox tests.
 
  I don't know about swift, but in neutron the intent was to have these
  tests be configurable to run against a real cloud, or not. Maru Newby
  would have details.
 
  I'd like to propose that tests of the third type get their own
  top-level package with a name of the form
  projectname-integrationtests (second choice: projectname-tempest
  on the principle that they're essentially plugins for Tempest). How
  would people feel about standardising that across OpenStack?
  +1 But I would not call it integrationtests for the reason given above.
 
 Because all heat does is interact with other services, what we call
 functional tests are actually integration tests. Sure, we could mock at
 the REST API level, but integration coverage is what we need most. This

I'd call that faking, not mocking, but both could apply.

 lets us verify things like:
 - how heat handles races in other services leading to resources going
 into ERROR

A fake that predictably fails (and thus tests failure handling) will
result in better coverage than a real service that only fails when that
real service is broken. What's frustrating is that _both_ are needed to
catch bugs.

 - connectivity and interaction between heat and agents on orchestrated
 servers
 

That is definitely