Re: [openstack-dev] [qa][all][Heat] Packaging of functional tests
On 04/09/14 10:45, Jay Pipes wrote: On 08/29/2014 05:15 PM, Zane Bitter wrote: On 29/08/14 14:27, Jay Pipes wrote: On 08/26/2014 10:14 AM, Zane Bitter wrote: Steve Baker has started the process of moving Heat tests out of the Tempest repository and into the Heat repository, and we're looking for some guidance on how they should be packaged in a consistent way. Apparently there are a few projects already packaging functional tests in the package projectname.tests.functional (alongside projectname.tests.unit for the unit tests). That strikes me as odd in our context, because while the unit tests run against the code in the package in which they are embedded, the functional tests run against some entirely different code - whatever OpenStack cloud you give it the auth URL and credentials for. So these tests run from the outside, just like their ancestors in Tempest do. There's all kinds of potential confusion here for users and packagers. None of it is fatal and all of it can be worked around, but if we refrain from doing the thing that makes zero conceptual sense then there will be no problem to work around :) I suspect from reading the previous thread about In-tree functional test vision that we may actually be dealing with three categories of test here rather than two: * Unit tests that run against the package they are embedded in * Functional tests that run against the package they are embedded in * Integration tests that run against a specified cloud i.e. the tests we are now trying to add to Heat might be qualitatively different from the projectname.tests.functional suites that already exist in a few projects. Perhaps someone from Neutron and/or Swift can confirm? I'd like to propose that tests of the third type get their own top-level package with a name of the form projectname-integrationtests (second choice: projectname-tempest on the principle that they're essentially plugins for Tempest). How would people feel about standardising that across OpenStack? By its nature, Heat is one of the only projects that would have integration tests of this nature. For Nova, there are some functional tests in nova/tests/integrated/ (yeah, badly named, I know) that are tests of the REST API endpoints and running service daemons (the things that are RPC endpoints), with a bunch of stuff faked out (like RPC comms, image services, authentication and the hypervisor layer itself). So, the integrated tests in Nova are really not testing integration with other projects, but rather integration of the subsystems and processes inside Nova. I'd support a policy that true integration tests -- tests that test the interaction between multiple real OpenStack service endpoints -- be left entirely to Tempest. Functional tests that test interaction between internal daemons and processes to a project should go into /$project/tests/functional/. For Heat, I believe tests that rely on faked-out other OpenStack services but stress the interaction between internal Heat daemons/processes should be in /heat/tests/functional/ and any tests the rely on working, real OpenStack service endpoints should be in Tempest. Well, the problem with that is that last time I checked there was exactly one Heat scenario test in Tempest because tempest-core doesn't have the bandwidth to merge all (any?) of the other ones folks submitted. So we're moving them to openstack/heat for the pure practical reason that it's the only way to get test coverage at all, rather than concerns about overloading the gate or theories about the best venue for cross-project integration testing. Hmm, speaking of passive aggressivity... That's probably a fair criticism, in light of what Matt said about the failures of communication on both sides. I think a formal liaison program will be an enormous help here. However, it won't change the fact that keeping the tests for every project in a single repo with a single core team just won't scale. Where can I see a discussion of the Heat integration tests with Tempest QA folks? If you give me some background on what efforts have been made already and what is remaining to be reviewed/merged/worked on, then I can try to get some resources dedicated to helping here. I made a list at one point: https://wiki.openstack.org/w/index.php?title=Governance/TechnicalCommittee/Heat_Gap_Coverageoldid=58358#Improved_functional_testing_with_Tempest I'm not sure how complete it is, because a lot of those patches came from folks who were not Heat core members, and in some cases not even closely engaged in Heat development. That wiki page was reviewed by the TC, although I was unable to make it to the meeting: http://eavesdrop.openstack.org/meetings/tc/2014/tc.2014-07-29-20.02.html I would greatly prefer just having a single source of integration testing in OpenStack, versus going back to the bad ol' days of everybody under the sun rewriting their own. I'd actually prefer some sort of plug-in system, where
Re: [openstack-dev] [qa][all][Heat] Packaging of functional tests
On 06/09/14 04:10, Matthew Treinish wrote: On Fri, Sep 05, 2014 at 09:42:17AM +1200, Steve Baker wrote: On 05/09/14 04:51, Matthew Treinish wrote: On Thu, Sep 04, 2014 at 04:32:53PM +0100, Steven Hardy wrote: On Thu, Sep 04, 2014 at 10:45:59AM -0400, Jay Pipes wrote: On 08/29/2014 05:15 PM, Zane Bitter wrote: On 29/08/14 14:27, Jay Pipes wrote: On 08/26/2014 10:14 AM, Zane Bitter wrote: Steve Baker has started the process of moving Heat tests out of the Tempest repository and into the Heat repository, and we're looking for some guidance on how they should be packaged in a consistent way. Apparently there are a few projects already packaging functional tests in the package projectname.tests.functional (alongside projectname.tests.unit for the unit tests). That strikes me as odd in our context, because while the unit tests run against the code in the package in which they are embedded, the functional tests run against some entirely different code - whatever OpenStack cloud you give it the auth URL and credentials for. So these tests run from the outside, just like their ancestors in Tempest do. There's all kinds of potential confusion here for users and packagers. None of it is fatal and all of it can be worked around, but if we refrain from doing the thing that makes zero conceptual sense then there will be no problem to work around :) I suspect from reading the previous thread about In-tree functional test vision that we may actually be dealing with three categories of test here rather than two: * Unit tests that run against the package they are embedded in * Functional tests that run against the package they are embedded in * Integration tests that run against a specified cloud i.e. the tests we are now trying to add to Heat might be qualitatively different from the projectname.tests.functional suites that already exist in a few projects. Perhaps someone from Neutron and/or Swift can confirm? I'd like to propose that tests of the third type get their own top-level package with a name of the form projectname-integrationtests (second choice: projectname-tempest on the principle that they're essentially plugins for Tempest). How would people feel about standardising that across OpenStack? By its nature, Heat is one of the only projects that would have integration tests of this nature. For Nova, there are some functional tests in nova/tests/integrated/ (yeah, badly named, I know) that are tests of the REST API endpoints and running service daemons (the things that are RPC endpoints), with a bunch of stuff faked out (like RPC comms, image services, authentication and the hypervisor layer itself). So, the integrated tests in Nova are really not testing integration with other projects, but rather integration of the subsystems and processes inside Nova. I'd support a policy that true integration tests -- tests that test the interaction between multiple real OpenStack service endpoints -- be left entirely to Tempest. Functional tests that test interaction between internal daemons and processes to a project should go into /$project/tests/functional/. For Heat, I believe tests that rely on faked-out other OpenStack services but stress the interaction between internal Heat daemons/processes should be in /heat/tests/functional/ and any tests the rely on working, real OpenStack service endpoints should be in Tempest. Well, the problem with that is that last time I checked there was exactly one Heat scenario test in Tempest because tempest-core doesn't have the bandwidth to merge all (any?) of the other ones folks submitted. So we're moving them to openstack/heat for the pure practical reason that it's the only way to get test coverage at all, rather than concerns about overloading the gate or theories about the best venue for cross-project integration testing. Hmm, speaking of passive aggressivity... Where can I see a discussion of the Heat integration tests with Tempest QA folks? If you give me some background on what efforts have been made already and what is remaining to be reviewed/merged/worked on, then I can try to get some resources dedicated to helping here. We recieved some fairly strong criticism from sdague[1] earlier this year, at which point we were already actively working on improving test coverage by writing new tests for tempest. Since then, several folks, myself included, commited very significant amounts of additional effort to writing more tests for tempest, with some success. Ultimately the review latency and overhead involved in constantly rebasing changes between infrequent reviews has resulted in slow progress and significant frustration for those attempting to contribute new test cases. It's been clear for a while that tempest-core have significant bandwidth issues, as well as not necessarily always having the specific domain expertise to thoroughly review some tests related to project-specific
Re: [openstack-dev] [qa][all][Heat] Packaging of functional tests
On Fri, Sep 05, 2014 at 09:42:17AM +1200, Steve Baker wrote: On 05/09/14 04:51, Matthew Treinish wrote: On Thu, Sep 04, 2014 at 04:32:53PM +0100, Steven Hardy wrote: On Thu, Sep 04, 2014 at 10:45:59AM -0400, Jay Pipes wrote: On 08/29/2014 05:15 PM, Zane Bitter wrote: On 29/08/14 14:27, Jay Pipes wrote: On 08/26/2014 10:14 AM, Zane Bitter wrote: Steve Baker has started the process of moving Heat tests out of the Tempest repository and into the Heat repository, and we're looking for some guidance on how they should be packaged in a consistent way. Apparently there are a few projects already packaging functional tests in the package projectname.tests.functional (alongside projectname.tests.unit for the unit tests). That strikes me as odd in our context, because while the unit tests run against the code in the package in which they are embedded, the functional tests run against some entirely different code - whatever OpenStack cloud you give it the auth URL and credentials for. So these tests run from the outside, just like their ancestors in Tempest do. There's all kinds of potential confusion here for users and packagers. None of it is fatal and all of it can be worked around, but if we refrain from doing the thing that makes zero conceptual sense then there will be no problem to work around :) I suspect from reading the previous thread about In-tree functional test vision that we may actually be dealing with three categories of test here rather than two: * Unit tests that run against the package they are embedded in * Functional tests that run against the package they are embedded in * Integration tests that run against a specified cloud i.e. the tests we are now trying to add to Heat might be qualitatively different from the projectname.tests.functional suites that already exist in a few projects. Perhaps someone from Neutron and/or Swift can confirm? I'd like to propose that tests of the third type get their own top-level package with a name of the form projectname-integrationtests (second choice: projectname-tempest on the principle that they're essentially plugins for Tempest). How would people feel about standardising that across OpenStack? By its nature, Heat is one of the only projects that would have integration tests of this nature. For Nova, there are some functional tests in nova/tests/integrated/ (yeah, badly named, I know) that are tests of the REST API endpoints and running service daemons (the things that are RPC endpoints), with a bunch of stuff faked out (like RPC comms, image services, authentication and the hypervisor layer itself). So, the integrated tests in Nova are really not testing integration with other projects, but rather integration of the subsystems and processes inside Nova. I'd support a policy that true integration tests -- tests that test the interaction between multiple real OpenStack service endpoints -- be left entirely to Tempest. Functional tests that test interaction between internal daemons and processes to a project should go into /$project/tests/functional/. For Heat, I believe tests that rely on faked-out other OpenStack services but stress the interaction between internal Heat daemons/processes should be in /heat/tests/functional/ and any tests the rely on working, real OpenStack service endpoints should be in Tempest. Well, the problem with that is that last time I checked there was exactly one Heat scenario test in Tempest because tempest-core doesn't have the bandwidth to merge all (any?) of the other ones folks submitted. So we're moving them to openstack/heat for the pure practical reason that it's the only way to get test coverage at all, rather than concerns about overloading the gate or theories about the best venue for cross-project integration testing. Hmm, speaking of passive aggressivity... Where can I see a discussion of the Heat integration tests with Tempest QA folks? If you give me some background on what efforts have been made already and what is remaining to be reviewed/merged/worked on, then I can try to get some resources dedicated to helping here. We recieved some fairly strong criticism from sdague[1] earlier this year, at which point we were already actively working on improving test coverage by writing new tests for tempest. Since then, several folks, myself included, commited very significant amounts of additional effort to writing more tests for tempest, with some success. Ultimately the review latency and overhead involved in constantly rebasing changes between infrequent reviews has resulted in slow progress and significant frustration for those attempting to contribute new test cases. It's been clear for a while that tempest-core have significant bandwidth issues, as well as not necessarily always having the specific domain expertise to
Re: [openstack-dev] [qa][all][Heat] Packaging of functional tests
On 09/05/2014 12:10 PM, Matthew Treinish wrote: On Fri, Sep 05, 2014 at 09:42:17AM +1200, Steve Baker wrote: On 05/09/14 04:51, Matthew Treinish wrote: On Thu, Sep 04, 2014 at 04:32:53PM +0100, Steven Hardy wrote: On Thu, Sep 04, 2014 at 10:45:59AM -0400, Jay Pipes wrote: On 08/29/2014 05:15 PM, Zane Bitter wrote: On 29/08/14 14:27, Jay Pipes wrote: On 08/26/2014 10:14 AM, Zane Bitter wrote: Steve Baker has started the process of moving Heat tests out of the Tempest repository and into the Heat repository, and we're looking for some guidance on how they should be packaged in a consistent way. Apparently there are a few projects already packaging functional tests in the package projectname.tests.functional (alongside projectname.tests.unit for the unit tests). That strikes me as odd in our context, because while the unit tests run against the code in the package in which they are embedded, the functional tests run against some entirely different code - whatever OpenStack cloud you give it the auth URL and credentials for. So these tests run from the outside, just like their ancestors in Tempest do. There's all kinds of potential confusion here for users and packagers. None of it is fatal and all of it can be worked around, but if we refrain from doing the thing that makes zero conceptual sense then there will be no problem to work around :) I suspect from reading the previous thread about In-tree functional test vision that we may actually be dealing with three categories of test here rather than two: * Unit tests that run against the package they are embedded in * Functional tests that run against the package they are embedded in * Integration tests that run against a specified cloud i.e. the tests we are now trying to add to Heat might be qualitatively different from the projectname.tests.functional suites that already exist in a few projects. Perhaps someone from Neutron and/or Swift can confirm? I'd like to propose that tests of the third type get their own top-level package with a name of the form projectname-integrationtests (second choice: projectname-tempest on the principle that they're essentially plugins for Tempest). How would people feel about standardising that across OpenStack? By its nature, Heat is one of the only projects that would have integration tests of this nature. For Nova, there are some functional tests in nova/tests/integrated/ (yeah, badly named, I know) that are tests of the REST API endpoints and running service daemons (the things that are RPC endpoints), with a bunch of stuff faked out (like RPC comms, image services, authentication and the hypervisor layer itself). So, the integrated tests in Nova are really not testing integration with other projects, but rather integration of the subsystems and processes inside Nova. I'd support a policy that true integration tests -- tests that test the interaction between multiple real OpenStack service endpoints -- be left entirely to Tempest. Functional tests that test interaction between internal daemons and processes to a project should go into /$project/tests/functional/. For Heat, I believe tests that rely on faked-out other OpenStack services but stress the interaction between internal Heat daemons/processes should be in /heat/tests/functional/ and any tests the rely on working, real OpenStack service endpoints should be in Tempest. Well, the problem with that is that last time I checked there was exactly one Heat scenario test in Tempest because tempest-core doesn't have the bandwidth to merge all (any?) of the other ones folks submitted. So we're moving them to openstack/heat for the pure practical reason that it's the only way to get test coverage at all, rather than concerns about overloading the gate or theories about the best venue for cross-project integration testing. Hmm, speaking of passive aggressivity... Where can I see a discussion of the Heat integration tests with Tempest QA folks? If you give me some background on what efforts have been made already and what is remaining to be reviewed/merged/worked on, then I can try to get some resources dedicated to helping here. We recieved some fairly strong criticism from sdague[1] earlier this year, at which point we were already actively working on improving test coverage by writing new tests for tempest. Since then, several folks, myself included, commited very significant amounts of additional effort to writing more tests for tempest, with some success. Ultimately the review latency and overhead involved in constantly rebasing changes between infrequent reviews has resulted in slow progress and significant frustration for those attempting to contribute new test cases. It's been clear for a while that tempest-core have significant bandwidth issues, as well as not necessarily always having the specific domain expertise to thoroughly review some tests related to project-specific behavior or functionality. So I view this as actually a breakdown
Re: [openstack-dev] [qa][all][Heat] Packaging of functional tests
On 08/29/2014 05:15 PM, Zane Bitter wrote: On 29/08/14 14:27, Jay Pipes wrote: On 08/26/2014 10:14 AM, Zane Bitter wrote: Steve Baker has started the process of moving Heat tests out of the Tempest repository and into the Heat repository, and we're looking for some guidance on how they should be packaged in a consistent way. Apparently there are a few projects already packaging functional tests in the package projectname.tests.functional (alongside projectname.tests.unit for the unit tests). That strikes me as odd in our context, because while the unit tests run against the code in the package in which they are embedded, the functional tests run against some entirely different code - whatever OpenStack cloud you give it the auth URL and credentials for. So these tests run from the outside, just like their ancestors in Tempest do. There's all kinds of potential confusion here for users and packagers. None of it is fatal and all of it can be worked around, but if we refrain from doing the thing that makes zero conceptual sense then there will be no problem to work around :) I suspect from reading the previous thread about In-tree functional test vision that we may actually be dealing with three categories of test here rather than two: * Unit tests that run against the package they are embedded in * Functional tests that run against the package they are embedded in * Integration tests that run against a specified cloud i.e. the tests we are now trying to add to Heat might be qualitatively different from the projectname.tests.functional suites that already exist in a few projects. Perhaps someone from Neutron and/or Swift can confirm? I'd like to propose that tests of the third type get their own top-level package with a name of the form projectname-integrationtests (second choice: projectname-tempest on the principle that they're essentially plugins for Tempest). How would people feel about standardising that across OpenStack? By its nature, Heat is one of the only projects that would have integration tests of this nature. For Nova, there are some functional tests in nova/tests/integrated/ (yeah, badly named, I know) that are tests of the REST API endpoints and running service daemons (the things that are RPC endpoints), with a bunch of stuff faked out (like RPC comms, image services, authentication and the hypervisor layer itself). So, the integrated tests in Nova are really not testing integration with other projects, but rather integration of the subsystems and processes inside Nova. I'd support a policy that true integration tests -- tests that test the interaction between multiple real OpenStack service endpoints -- be left entirely to Tempest. Functional tests that test interaction between internal daemons and processes to a project should go into /$project/tests/functional/. For Heat, I believe tests that rely on faked-out other OpenStack services but stress the interaction between internal Heat daemons/processes should be in /heat/tests/functional/ and any tests the rely on working, real OpenStack service endpoints should be in Tempest. Well, the problem with that is that last time I checked there was exactly one Heat scenario test in Tempest because tempest-core doesn't have the bandwidth to merge all (any?) of the other ones folks submitted. So we're moving them to openstack/heat for the pure practical reason that it's the only way to get test coverage at all, rather than concerns about overloading the gate or theories about the best venue for cross-project integration testing. Hmm, speaking of passive aggressivity... Where can I see a discussion of the Heat integration tests with Tempest QA folks? If you give me some background on what efforts have been made already and what is remaining to be reviewed/merged/worked on, then I can try to get some resources dedicated to helping here. I would greatly prefer just having a single source of integration testing in OpenStack, versus going back to the bad ol' days of everybody under the sun rewriting their own. Note that I'm not talking about functional testing here, just the integration testing... Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][all][Heat] Packaging of functional tests
On Thu, Sep 04, 2014 at 10:45:59AM -0400, Jay Pipes wrote: On 08/29/2014 05:15 PM, Zane Bitter wrote: On 29/08/14 14:27, Jay Pipes wrote: On 08/26/2014 10:14 AM, Zane Bitter wrote: Steve Baker has started the process of moving Heat tests out of the Tempest repository and into the Heat repository, and we're looking for some guidance on how they should be packaged in a consistent way. Apparently there are a few projects already packaging functional tests in the package projectname.tests.functional (alongside projectname.tests.unit for the unit tests). That strikes me as odd in our context, because while the unit tests run against the code in the package in which they are embedded, the functional tests run against some entirely different code - whatever OpenStack cloud you give it the auth URL and credentials for. So these tests run from the outside, just like their ancestors in Tempest do. There's all kinds of potential confusion here for users and packagers. None of it is fatal and all of it can be worked around, but if we refrain from doing the thing that makes zero conceptual sense then there will be no problem to work around :) I suspect from reading the previous thread about In-tree functional test vision that we may actually be dealing with three categories of test here rather than two: * Unit tests that run against the package they are embedded in * Functional tests that run against the package they are embedded in * Integration tests that run against a specified cloud i.e. the tests we are now trying to add to Heat might be qualitatively different from the projectname.tests.functional suites that already exist in a few projects. Perhaps someone from Neutron and/or Swift can confirm? I'd like to propose that tests of the third type get their own top-level package with a name of the form projectname-integrationtests (second choice: projectname-tempest on the principle that they're essentially plugins for Tempest). How would people feel about standardising that across OpenStack? By its nature, Heat is one of the only projects that would have integration tests of this nature. For Nova, there are some functional tests in nova/tests/integrated/ (yeah, badly named, I know) that are tests of the REST API endpoints and running service daemons (the things that are RPC endpoints), with a bunch of stuff faked out (like RPC comms, image services, authentication and the hypervisor layer itself). So, the integrated tests in Nova are really not testing integration with other projects, but rather integration of the subsystems and processes inside Nova. I'd support a policy that true integration tests -- tests that test the interaction between multiple real OpenStack service endpoints -- be left entirely to Tempest. Functional tests that test interaction between internal daemons and processes to a project should go into /$project/tests/functional/. For Heat, I believe tests that rely on faked-out other OpenStack services but stress the interaction between internal Heat daemons/processes should be in /heat/tests/functional/ and any tests the rely on working, real OpenStack service endpoints should be in Tempest. Well, the problem with that is that last time I checked there was exactly one Heat scenario test in Tempest because tempest-core doesn't have the bandwidth to merge all (any?) of the other ones folks submitted. So we're moving them to openstack/heat for the pure practical reason that it's the only way to get test coverage at all, rather than concerns about overloading the gate or theories about the best venue for cross-project integration testing. Hmm, speaking of passive aggressivity... Where can I see a discussion of the Heat integration tests with Tempest QA folks? If you give me some background on what efforts have been made already and what is remaining to be reviewed/merged/worked on, then I can try to get some resources dedicated to helping here. We recieved some fairly strong criticism from sdague[1] earlier this year, at which point we were already actively working on improving test coverage by writing new tests for tempest. Since then, several folks, myself included, commited very significant amounts of additional effort to writing more tests for tempest, with some success. Ultimately the review latency and overhead involved in constantly rebasing changes between infrequent reviews has resulted in slow progress and significant frustration for those attempting to contribute new test cases. It's been clear for a while that tempest-core have significant bandwidth issues, as well as not necessarily always having the specific domain expertise to thoroughly review some tests related to project-specific behavior or functionality. So it was with some relief that we saw the proposal[2] to move the burden for reviewing project test-cases to the project teams, who will presumably be more motivated to do the reviews,
Re: [openstack-dev] [qa][all][Heat] Packaging of functional tests
On 09/04/2014 11:32 AM, Steven Hardy wrote: On Thu, Sep 04, 2014 at 10:45:59AM -0400, Jay Pipes wrote: On 08/29/2014 05:15 PM, Zane Bitter wrote: On 29/08/14 14:27, Jay Pipes wrote: On 08/26/2014 10:14 AM, Zane Bitter wrote: Steve Baker has started the process of moving Heat tests out of the Tempest repository and into the Heat repository, and we're looking for some guidance on how they should be packaged in a consistent way. Apparently there are a few projects already packaging functional tests in the package projectname.tests.functional (alongside projectname.tests.unit for the unit tests). That strikes me as odd in our context, because while the unit tests run against the code in the package in which they are embedded, the functional tests run against some entirely different code - whatever OpenStack cloud you give it the auth URL and credentials for. So these tests run from the outside, just like their ancestors in Tempest do. There's all kinds of potential confusion here for users and packagers. None of it is fatal and all of it can be worked around, but if we refrain from doing the thing that makes zero conceptual sense then there will be no problem to work around :) I suspect from reading the previous thread about In-tree functional test vision that we may actually be dealing with three categories of test here rather than two: * Unit tests that run against the package they are embedded in * Functional tests that run against the package they are embedded in * Integration tests that run against a specified cloud i.e. the tests we are now trying to add to Heat might be qualitatively different from the projectname.tests.functional suites that already exist in a few projects. Perhaps someone from Neutron and/or Swift can confirm? I'd like to propose that tests of the third type get their own top-level package with a name of the form projectname-integrationtests (second choice: projectname-tempest on the principle that they're essentially plugins for Tempest). How would people feel about standardising that across OpenStack? By its nature, Heat is one of the only projects that would have integration tests of this nature. For Nova, there are some functional tests in nova/tests/integrated/ (yeah, badly named, I know) that are tests of the REST API endpoints and running service daemons (the things that are RPC endpoints), with a bunch of stuff faked out (like RPC comms, image services, authentication and the hypervisor layer itself). So, the integrated tests in Nova are really not testing integration with other projects, but rather integration of the subsystems and processes inside Nova. I'd support a policy that true integration tests -- tests that test the interaction between multiple real OpenStack service endpoints -- be left entirely to Tempest. Functional tests that test interaction between internal daemons and processes to a project should go into /$project/tests/functional/. For Heat, I believe tests that rely on faked-out other OpenStack services but stress the interaction between internal Heat daemons/processes should be in /heat/tests/functional/ and any tests the rely on working, real OpenStack service endpoints should be in Tempest. Well, the problem with that is that last time I checked there was exactly one Heat scenario test in Tempest because tempest-core doesn't have the bandwidth to merge all (any?) of the other ones folks submitted. So we're moving them to openstack/heat for the pure practical reason that it's the only way to get test coverage at all, rather than concerns about overloading the gate or theories about the best venue for cross-project integration testing. Hmm, speaking of passive aggressivity... Where can I see a discussion of the Heat integration tests with Tempest QA folks? If you give me some background on what efforts have been made already and what is remaining to be reviewed/merged/worked on, then I can try to get some resources dedicated to helping here. We recieved some fairly strong criticism from sdague[1] earlier this year, at which point we were already actively working on improving test coverage by writing new tests for tempest. Since then, several folks, myself included, commited very significant amounts of additional effort to writing more tests for tempest, with some success. Ultimately the review latency and overhead involved in constantly rebasing changes between infrequent reviews has resulted in slow progress and significant frustration for those attempting to contribute new test cases. It's been clear for a while that tempest-core have significant bandwidth issues, as well as not necessarily always having the specific domain expertise to thoroughly review some tests related to project-specific behavior or functionality. So it was with some relief that we saw the proposal[2] to move the burden for reviewing project test-cases to the project teams, who will presumably be more motivated to do the reviews, and have the knowledge
Re: [openstack-dev] [qa][all][Heat] Packaging of functional tests
On Thu, Sep 04, 2014 at 04:32:53PM +0100, Steven Hardy wrote: On Thu, Sep 04, 2014 at 10:45:59AM -0400, Jay Pipes wrote: On 08/29/2014 05:15 PM, Zane Bitter wrote: On 29/08/14 14:27, Jay Pipes wrote: On 08/26/2014 10:14 AM, Zane Bitter wrote: Steve Baker has started the process of moving Heat tests out of the Tempest repository and into the Heat repository, and we're looking for some guidance on how they should be packaged in a consistent way. Apparently there are a few projects already packaging functional tests in the package projectname.tests.functional (alongside projectname.tests.unit for the unit tests). That strikes me as odd in our context, because while the unit tests run against the code in the package in which they are embedded, the functional tests run against some entirely different code - whatever OpenStack cloud you give it the auth URL and credentials for. So these tests run from the outside, just like their ancestors in Tempest do. There's all kinds of potential confusion here for users and packagers. None of it is fatal and all of it can be worked around, but if we refrain from doing the thing that makes zero conceptual sense then there will be no problem to work around :) I suspect from reading the previous thread about In-tree functional test vision that we may actually be dealing with three categories of test here rather than two: * Unit tests that run against the package they are embedded in * Functional tests that run against the package they are embedded in * Integration tests that run against a specified cloud i.e. the tests we are now trying to add to Heat might be qualitatively different from the projectname.tests.functional suites that already exist in a few projects. Perhaps someone from Neutron and/or Swift can confirm? I'd like to propose that tests of the third type get their own top-level package with a name of the form projectname-integrationtests (second choice: projectname-tempest on the principle that they're essentially plugins for Tempest). How would people feel about standardising that across OpenStack? By its nature, Heat is one of the only projects that would have integration tests of this nature. For Nova, there are some functional tests in nova/tests/integrated/ (yeah, badly named, I know) that are tests of the REST API endpoints and running service daemons (the things that are RPC endpoints), with a bunch of stuff faked out (like RPC comms, image services, authentication and the hypervisor layer itself). So, the integrated tests in Nova are really not testing integration with other projects, but rather integration of the subsystems and processes inside Nova. I'd support a policy that true integration tests -- tests that test the interaction between multiple real OpenStack service endpoints -- be left entirely to Tempest. Functional tests that test interaction between internal daemons and processes to a project should go into /$project/tests/functional/. For Heat, I believe tests that rely on faked-out other OpenStack services but stress the interaction between internal Heat daemons/processes should be in /heat/tests/functional/ and any tests the rely on working, real OpenStack service endpoints should be in Tempest. Well, the problem with that is that last time I checked there was exactly one Heat scenario test in Tempest because tempest-core doesn't have the bandwidth to merge all (any?) of the other ones folks submitted. So we're moving them to openstack/heat for the pure practical reason that it's the only way to get test coverage at all, rather than concerns about overloading the gate or theories about the best venue for cross-project integration testing. Hmm, speaking of passive aggressivity... Where can I see a discussion of the Heat integration tests with Tempest QA folks? If you give me some background on what efforts have been made already and what is remaining to be reviewed/merged/worked on, then I can try to get some resources dedicated to helping here. We recieved some fairly strong criticism from sdague[1] earlier this year, at which point we were already actively working on improving test coverage by writing new tests for tempest. Since then, several folks, myself included, commited very significant amounts of additional effort to writing more tests for tempest, with some success. Ultimately the review latency and overhead involved in constantly rebasing changes between infrequent reviews has resulted in slow progress and significant frustration for those attempting to contribute new test cases. It's been clear for a while that tempest-core have significant bandwidth issues, as well as not necessarily always having the specific domain expertise to thoroughly review some tests related to project-specific behavior or functionality. So I view this as actually a
Re: [openstack-dev] [qa][all][Heat] Packaging of functional tests
On 05/09/14 04:51, Matthew Treinish wrote: On Thu, Sep 04, 2014 at 04:32:53PM +0100, Steven Hardy wrote: On Thu, Sep 04, 2014 at 10:45:59AM -0400, Jay Pipes wrote: On 08/29/2014 05:15 PM, Zane Bitter wrote: On 29/08/14 14:27, Jay Pipes wrote: On 08/26/2014 10:14 AM, Zane Bitter wrote: Steve Baker has started the process of moving Heat tests out of the Tempest repository and into the Heat repository, and we're looking for some guidance on how they should be packaged in a consistent way. Apparently there are a few projects already packaging functional tests in the package projectname.tests.functional (alongside projectname.tests.unit for the unit tests). That strikes me as odd in our context, because while the unit tests run against the code in the package in which they are embedded, the functional tests run against some entirely different code - whatever OpenStack cloud you give it the auth URL and credentials for. So these tests run from the outside, just like their ancestors in Tempest do. There's all kinds of potential confusion here for users and packagers. None of it is fatal and all of it can be worked around, but if we refrain from doing the thing that makes zero conceptual sense then there will be no problem to work around :) I suspect from reading the previous thread about In-tree functional test vision that we may actually be dealing with three categories of test here rather than two: * Unit tests that run against the package they are embedded in * Functional tests that run against the package they are embedded in * Integration tests that run against a specified cloud i.e. the tests we are now trying to add to Heat might be qualitatively different from the projectname.tests.functional suites that already exist in a few projects. Perhaps someone from Neutron and/or Swift can confirm? I'd like to propose that tests of the third type get their own top-level package with a name of the form projectname-integrationtests (second choice: projectname-tempest on the principle that they're essentially plugins for Tempest). How would people feel about standardising that across OpenStack? By its nature, Heat is one of the only projects that would have integration tests of this nature. For Nova, there are some functional tests in nova/tests/integrated/ (yeah, badly named, I know) that are tests of the REST API endpoints and running service daemons (the things that are RPC endpoints), with a bunch of stuff faked out (like RPC comms, image services, authentication and the hypervisor layer itself). So, the integrated tests in Nova are really not testing integration with other projects, but rather integration of the subsystems and processes inside Nova. I'd support a policy that true integration tests -- tests that test the interaction between multiple real OpenStack service endpoints -- be left entirely to Tempest. Functional tests that test interaction between internal daemons and processes to a project should go into /$project/tests/functional/. For Heat, I believe tests that rely on faked-out other OpenStack services but stress the interaction between internal Heat daemons/processes should be in /heat/tests/functional/ and any tests the rely on working, real OpenStack service endpoints should be in Tempest. Well, the problem with that is that last time I checked there was exactly one Heat scenario test in Tempest because tempest-core doesn't have the bandwidth to merge all (any?) of the other ones folks submitted. So we're moving them to openstack/heat for the pure practical reason that it's the only way to get test coverage at all, rather than concerns about overloading the gate or theories about the best venue for cross-project integration testing. Hmm, speaking of passive aggressivity... Where can I see a discussion of the Heat integration tests with Tempest QA folks? If you give me some background on what efforts have been made already and what is remaining to be reviewed/merged/worked on, then I can try to get some resources dedicated to helping here. We recieved some fairly strong criticism from sdague[1] earlier this year, at which point we were already actively working on improving test coverage by writing new tests for tempest. Since then, several folks, myself included, commited very significant amounts of additional effort to writing more tests for tempest, with some success. Ultimately the review latency and overhead involved in constantly rebasing changes between infrequent reviews has resulted in slow progress and significant frustration for those attempting to contribute new test cases. It's been clear for a while that tempest-core have significant bandwidth issues, as well as not necessarily always having the specific domain expertise to thoroughly review some tests related to project-specific behavior or functionality. So I view this as actually a breakdown in cross-team communication, with both sides
Re: [openstack-dev] [qa][all][Heat] Packaging of functional tests
On Fri, Sep 5, 2014 at 2:40 AM, Jay Pipes jaypi...@gmail.com wrote: On 09/04/2014 11:32 AM, Steven Hardy wrote: On Thu, Sep 04, 2014 at 10:45:59AM -0400, Jay Pipes wrote: On 08/29/2014 05:15 PM, Zane Bitter wrote: On 29/08/14 14:27, Jay Pipes wrote: On 08/26/2014 10:14 AM, Zane Bitter wrote: Steve Baker has started the process of moving Heat tests out of the Tempest repository and into the Heat repository, and we're looking for some guidance on how they should be packaged in a consistent way. Apparently there are a few projects already packaging functional tests in the package projectname.tests.functional (alongside projectname.tests.unit for the unit tests). That strikes me as odd in our context, because while the unit tests run against the code in the package in which they are embedded, the functional tests run against some entirely different code - whatever OpenStack cloud you give it the auth URL and credentials for. So these tests run from the outside, just like their ancestors in Tempest do. There's all kinds of potential confusion here for users and packagers. None of it is fatal and all of it can be worked around, but if we refrain from doing the thing that makes zero conceptual sense then there will be no problem to work around :) I suspect from reading the previous thread about In-tree functional test vision that we may actually be dealing with three categories of test here rather than two: * Unit tests that run against the package they are embedded in * Functional tests that run against the package they are embedded in * Integration tests that run against a specified cloud i.e. the tests we are now trying to add to Heat might be qualitatively different from the projectname.tests.functional suites that already exist in a few projects. Perhaps someone from Neutron and/or Swift can confirm? I'd like to propose that tests of the third type get their own top-level package with a name of the form projectname-integrationtests (second choice: projectname-tempest on the principle that they're essentially plugins for Tempest). How would people feel about standardising that across OpenStack? By its nature, Heat is one of the only projects that would have integration tests of this nature. For Nova, there are some functional tests in nova/tests/integrated/ (yeah, badly named, I know) that are tests of the REST API endpoints and running service daemons (the things that are RPC endpoints), with a bunch of stuff faked out (like RPC comms, image services, authentication and the hypervisor layer itself). So, the integrated tests in Nova are really not testing integration with other projects, but rather integration of the subsystems and processes inside Nova. I'd support a policy that true integration tests -- tests that test the interaction between multiple real OpenStack service endpoints -- be left entirely to Tempest. Functional tests that test interaction between internal daemons and processes to a project should go into /$project/tests/functional/. For Heat, I believe tests that rely on faked-out other OpenStack services but stress the interaction between internal Heat daemons/processes should be in /heat/tests/functional/ and any tests the rely on working, real OpenStack service endpoints should be in Tempest. Well, the problem with that is that last time I checked there was exactly one Heat scenario test in Tempest because tempest-core doesn't have the bandwidth to merge all (any?) of the other ones folks submitted. So we're moving them to openstack/heat for the pure practical reason that it's the only way to get test coverage at all, rather than concerns about overloading the gate or theories about the best venue for cross-project integration testing. Hmm, speaking of passive aggressivity... Where can I see a discussion of the Heat integration tests with Tempest QA folks? If you give me some background on what efforts have been made already and what is remaining to be reviewed/merged/worked on, then I can try to get some resources dedicated to helping here. We recieved some fairly strong criticism from sdague[1] earlier this year, at which point we were already actively working on improving test coverage by writing new tests for tempest. Since then, several folks, myself included, commited very significant amounts of additional effort to writing more tests for tempest, with some success. Ultimately the review latency and overhead involved in constantly rebasing changes between infrequent reviews has resulted in slow progress and significant frustration for those attempting to contribute new test cases. It's been clear for a while that tempest-core have significant bandwidth issues, as well as not necessarily always having the specific domain expertise to thoroughly review some tests related to project-specific behavior or functionality. So it was with some relief that we saw the
Re: [openstack-dev] [qa][all][Heat] Packaging of functional tests
On 29/08/14 14:27, Jay Pipes wrote: On 08/26/2014 10:14 AM, Zane Bitter wrote: Steve Baker has started the process of moving Heat tests out of the Tempest repository and into the Heat repository, and we're looking for some guidance on how they should be packaged in a consistent way. Apparently there are a few projects already packaging functional tests in the package projectname.tests.functional (alongside projectname.tests.unit for the unit tests). That strikes me as odd in our context, because while the unit tests run against the code in the package in which they are embedded, the functional tests run against some entirely different code - whatever OpenStack cloud you give it the auth URL and credentials for. So these tests run from the outside, just like their ancestors in Tempest do. There's all kinds of potential confusion here for users and packagers. None of it is fatal and all of it can be worked around, but if we refrain from doing the thing that makes zero conceptual sense then there will be no problem to work around :) I suspect from reading the previous thread about In-tree functional test vision that we may actually be dealing with three categories of test here rather than two: * Unit tests that run against the package they are embedded in * Functional tests that run against the package they are embedded in * Integration tests that run against a specified cloud i.e. the tests we are now trying to add to Heat might be qualitatively different from the projectname.tests.functional suites that already exist in a few projects. Perhaps someone from Neutron and/or Swift can confirm? I'd like to propose that tests of the third type get their own top-level package with a name of the form projectname-integrationtests (second choice: projectname-tempest on the principle that they're essentially plugins for Tempest). How would people feel about standardising that across OpenStack? By its nature, Heat is one of the only projects that would have integration tests of this nature. For Nova, there are some functional tests in nova/tests/integrated/ (yeah, badly named, I know) that are tests of the REST API endpoints and running service daemons (the things that are RPC endpoints), with a bunch of stuff faked out (like RPC comms, image services, authentication and the hypervisor layer itself). So, the integrated tests in Nova are really not testing integration with other projects, but rather integration of the subsystems and processes inside Nova. I'd support a policy that true integration tests -- tests that test the interaction between multiple real OpenStack service endpoints -- be left entirely to Tempest. Functional tests that test interaction between internal daemons and processes to a project should go into /$project/tests/functional/. For Heat, I believe tests that rely on faked-out other OpenStack services but stress the interaction between internal Heat daemons/processes should be in /heat/tests/functional/ and any tests the rely on working, real OpenStack service endpoints should be in Tempest. Well, the problem with that is that last time I checked there was exactly one Heat scenario test in Tempest because tempest-core doesn't have the bandwidth to merge all (any?) of the other ones folks submitted. So we're moving them to openstack/heat for the pure practical reason that it's the only way to get test coverage at all, rather than concerns about overloading the gate or theories about the best venue for cross-project integration testing. cheers, Zane. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][all][Heat] Packaging of functional tests
On 26/08/14 18:59, Clint Byrum wrote: Excerpts from Steve Baker's message of 2014-08-26 14:25:46 -0700: On 27/08/14 03:18, David Kranz wrote: On 08/26/2014 10:14 AM, Zane Bitter wrote: Steve Baker has started the process of moving Heat tests out of the Tempest repository and into the Heat repository, and we're looking for some guidance on how they should be packaged in a consistent way. Apparently there are a few projects already packaging functional tests in the package projectname.tests.functional (alongside projectname.tests.unit for the unit tests). That strikes me as odd in our context, because while the unit tests run against the code in the package in which they are embedded, the functional tests run against some entirely different code - whatever OpenStack cloud you give it the auth URL and credentials for. So these tests run from the outside, just like their ancestors in Tempest do. There's all kinds of potential confusion here for users and packagers. None of it is fatal and all of it can be worked around, but if we refrain from doing the thing that makes zero conceptual sense then there will be no problem to work around :) Thanks, Zane. The point of moving functional tests to projects is to be able to run more of them in gate jobs for those projects, and allow tempest to survive being stretched-to-breaking horizontally as we scale to more projects. At the same time, there are benefits to the tempest-as-all-in-one-functional-and-integration-suite that we should try not to lose: 1. Strong integration testing without thinking too hard about the actual dependencies 2. Protection from mistaken or unwise api changes (tempest two-step required) 3. Exportability as a complete blackbox functional test suite that can be used by Rally, RefStack, deployment validation, etc. I think (1) may be the most challenging because tests that are moved out of tempest might be testing some integration that is not being covered by a scenario. We will need to make sure that tempest actually has a complete enough set of tests to validate integration. Even if this is all implemented in a way where tempest can see in-project tests as plugins, there will still not be time to run them all as part of tempest on every commit to every project, so a selection will have to be made. (2) is quite difficult. In Atlanta we talked about taking a copy of functional tests into tempest for stable apis. I don't know how workable that is but don't see any other real options except vigilance in reviews of patches that change functional tests. (3) is what Zane was addressing. The in-project functional tests need to be written in a way that they can, at least in some configuration, run against a real cloud. I suspect from reading the previous thread about In-tree functional test vision that we may actually be dealing with three categories of test here rather than two: * Unit tests that run against the package they are embedded in * Functional tests that run against the package they are embedded in * Integration tests that run against a specified cloud i.e. the tests we are now trying to add to Heat might be qualitatively different from the projectname.tests.functional suites that already exist in a few projects. Perhaps someone from Neutron and/or Swift can confirm? That seems right, except that I would call the third functional tests and not integration tests, because the purpose is not really integration but deep testing of a particular service. Tempest would continue to focus on integration testing. Is there some controversy about that? The second category could include whitebox tests. I don't know about swift, but in neutron the intent was to have these tests be configurable to run against a real cloud, or not. Maru Newby would have details. I'd like to propose that tests of the third type get their own top-level package with a name of the form projectname-integrationtests (second choice: projectname-tempest on the principle that they're essentially plugins for Tempest). How would people feel about standardising that across OpenStack? +1 But I would not call it integrationtests for the reason given above. Because all heat does is interact with other services, what we call functional tests are actually integration tests. Sure, we could mock at the REST API level, but integration coverage is what we need most. This I'd call that faking, not mocking, but both could apply. lets us verify things like: - how heat handles races in other services leading to resources going into ERROR A fake that predictably fails (and thus tests failure handling) will result in better coverage than a real service that only fails when that real service is broken. What's frustrating is that _both_ are needed to catch bugs. Yeah, we discussed this in the project meeting yesterday[1] and basically came to the conclusion that we'll ultimately need both functional tests with fake services and integration tests with real services.
[openstack-dev] [qa][all][Heat] Packaging of functional tests
Steve Baker has started the process of moving Heat tests out of the Tempest repository and into the Heat repository, and we're looking for some guidance on how they should be packaged in a consistent way. Apparently there are a few projects already packaging functional tests in the package projectname.tests.functional (alongside projectname.tests.unit for the unit tests). That strikes me as odd in our context, because while the unit tests run against the code in the package in which they are embedded, the functional tests run against some entirely different code - whatever OpenStack cloud you give it the auth URL and credentials for. So these tests run from the outside, just like their ancestors in Tempest do. There's all kinds of potential confusion here for users and packagers. None of it is fatal and all of it can be worked around, but if we refrain from doing the thing that makes zero conceptual sense then there will be no problem to work around :) I suspect from reading the previous thread about In-tree functional test vision that we may actually be dealing with three categories of test here rather than two: * Unit tests that run against the package they are embedded in * Functional tests that run against the package they are embedded in * Integration tests that run against a specified cloud i.e. the tests we are now trying to add to Heat might be qualitatively different from the projectname.tests.functional suites that already exist in a few projects. Perhaps someone from Neutron and/or Swift can confirm? I'd like to propose that tests of the third type get their own top-level package with a name of the form projectname-integrationtests (second choice: projectname-tempest on the principle that they're essentially plugins for Tempest). How would people feel about standardising that across OpenStack? thanks, Zane. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][all][Heat] Packaging of functional tests
On 08/26/2014 10:14 AM, Zane Bitter wrote: Steve Baker has started the process of moving Heat tests out of the Tempest repository and into the Heat repository, and we're looking for some guidance on how they should be packaged in a consistent way. Apparently there are a few projects already packaging functional tests in the package projectname.tests.functional (alongside projectname.tests.unit for the unit tests). That strikes me as odd in our context, because while the unit tests run against the code in the package in which they are embedded, the functional tests run against some entirely different code - whatever OpenStack cloud you give it the auth URL and credentials for. So these tests run from the outside, just like their ancestors in Tempest do. There's all kinds of potential confusion here for users and packagers. None of it is fatal and all of it can be worked around, but if we refrain from doing the thing that makes zero conceptual sense then there will be no problem to work around :) Thanks, Zane. The point of moving functional tests to projects is to be able to run more of them in gate jobs for those projects, and allow tempest to survive being stretched-to-breaking horizontally as we scale to more projects. At the same time, there are benefits to the tempest-as-all-in-one-functional-and-integration-suite that we should try not to lose: 1. Strong integration testing without thinking too hard about the actual dependencies 2. Protection from mistaken or unwise api changes (tempest two-step required) 3. Exportability as a complete blackbox functional test suite that can be used by Rally, RefStack, deployment validation, etc. I think (1) may be the most challenging because tests that are moved out of tempest might be testing some integration that is not being covered by a scenario. We will need to make sure that tempest actually has a complete enough set of tests to validate integration. Even if this is all implemented in a way where tempest can see in-project tests as plugins, there will still not be time to run them all as part of tempest on every commit to every project, so a selection will have to be made. (2) is quite difficult. In Atlanta we talked about taking a copy of functional tests into tempest for stable apis. I don't know how workable that is but don't see any other real options except vigilance in reviews of patches that change functional tests. (3) is what Zane was addressing. The in-project functional tests need to be written in a way that they can, at least in some configuration, run against a real cloud. I suspect from reading the previous thread about In-tree functional test vision that we may actually be dealing with three categories of test here rather than two: * Unit tests that run against the package they are embedded in * Functional tests that run against the package they are embedded in * Integration tests that run against a specified cloud i.e. the tests we are now trying to add to Heat might be qualitatively different from the projectname.tests.functional suites that already exist in a few projects. Perhaps someone from Neutron and/or Swift can confirm? That seems right, except that I would call the third functional tests and not integration tests, because the purpose is not really integration but deep testing of a particular service. Tempest would continue to focus on integration testing. Is there some controversy about that? The second category could include whitebox tests. I don't know about swift, but in neutron the intent was to have these tests be configurable to run against a real cloud, or not. Maru Newby would have details. I'd like to propose that tests of the third type get their own top-level package with a name of the form projectname-integrationtests (second choice: projectname-tempest on the principle that they're essentially plugins for Tempest). How would people feel about standardising that across OpenStack? +1 But I would not call it integrationtests for the reason given above. -David thanks, Zane. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][all][Heat] Packaging of functional tests
On 27/08/14 03:18, David Kranz wrote: On 08/26/2014 10:14 AM, Zane Bitter wrote: Steve Baker has started the process of moving Heat tests out of the Tempest repository and into the Heat repository, and we're looking for some guidance on how they should be packaged in a consistent way. Apparently there are a few projects already packaging functional tests in the package projectname.tests.functional (alongside projectname.tests.unit for the unit tests). That strikes me as odd in our context, because while the unit tests run against the code in the package in which they are embedded, the functional tests run against some entirely different code - whatever OpenStack cloud you give it the auth URL and credentials for. So these tests run from the outside, just like their ancestors in Tempest do. There's all kinds of potential confusion here for users and packagers. None of it is fatal and all of it can be worked around, but if we refrain from doing the thing that makes zero conceptual sense then there will be no problem to work around :) Thanks, Zane. The point of moving functional tests to projects is to be able to run more of them in gate jobs for those projects, and allow tempest to survive being stretched-to-breaking horizontally as we scale to more projects. At the same time, there are benefits to the tempest-as-all-in-one-functional-and-integration-suite that we should try not to lose: 1. Strong integration testing without thinking too hard about the actual dependencies 2. Protection from mistaken or unwise api changes (tempest two-step required) 3. Exportability as a complete blackbox functional test suite that can be used by Rally, RefStack, deployment validation, etc. I think (1) may be the most challenging because tests that are moved out of tempest might be testing some integration that is not being covered by a scenario. We will need to make sure that tempest actually has a complete enough set of tests to validate integration. Even if this is all implemented in a way where tempest can see in-project tests as plugins, there will still not be time to run them all as part of tempest on every commit to every project, so a selection will have to be made. (2) is quite difficult. In Atlanta we talked about taking a copy of functional tests into tempest for stable apis. I don't know how workable that is but don't see any other real options except vigilance in reviews of patches that change functional tests. (3) is what Zane was addressing. The in-project functional tests need to be written in a way that they can, at least in some configuration, run against a real cloud. I suspect from reading the previous thread about In-tree functional test vision that we may actually be dealing with three categories of test here rather than two: * Unit tests that run against the package they are embedded in * Functional tests that run against the package they are embedded in * Integration tests that run against a specified cloud i.e. the tests we are now trying to add to Heat might be qualitatively different from the projectname.tests.functional suites that already exist in a few projects. Perhaps someone from Neutron and/or Swift can confirm? That seems right, except that I would call the third functional tests and not integration tests, because the purpose is not really integration but deep testing of a particular service. Tempest would continue to focus on integration testing. Is there some controversy about that? The second category could include whitebox tests. I don't know about swift, but in neutron the intent was to have these tests be configurable to run against a real cloud, or not. Maru Newby would have details. I'd like to propose that tests of the third type get their own top-level package with a name of the form projectname-integrationtests (second choice: projectname-tempest on the principle that they're essentially plugins for Tempest). How would people feel about standardising that across OpenStack? +1 But I would not call it integrationtests for the reason given above. Because all heat does is interact with other services, what we call functional tests are actually integration tests. Sure, we could mock at the REST API level, but integration coverage is what we need most. This lets us verify things like: - how heat handles races in other services leading to resources going into ERROR - connectivity and interaction between heat and agents on orchestrated servers So I'm +1 on Zane's suggestion for projectname-integrationtests for the sort of scenario integration testing we want to do. I would imagine the swift and neutron functional tests are much more self-contained, possibly only interacting with keystone. The heat API tests are very much in this category, which currently live in tempest.api.orchestration. These can move into the heat tree using whatever established convention once tempest-lib is consumable, and/or
Re: [openstack-dev] [qa][all][Heat] Packaging of functional tests
Excerpts from Steve Baker's message of 2014-08-26 14:25:46 -0700: On 27/08/14 03:18, David Kranz wrote: On 08/26/2014 10:14 AM, Zane Bitter wrote: Steve Baker has started the process of moving Heat tests out of the Tempest repository and into the Heat repository, and we're looking for some guidance on how they should be packaged in a consistent way. Apparently there are a few projects already packaging functional tests in the package projectname.tests.functional (alongside projectname.tests.unit for the unit tests). That strikes me as odd in our context, because while the unit tests run against the code in the package in which they are embedded, the functional tests run against some entirely different code - whatever OpenStack cloud you give it the auth URL and credentials for. So these tests run from the outside, just like their ancestors in Tempest do. There's all kinds of potential confusion here for users and packagers. None of it is fatal and all of it can be worked around, but if we refrain from doing the thing that makes zero conceptual sense then there will be no problem to work around :) Thanks, Zane. The point of moving functional tests to projects is to be able to run more of them in gate jobs for those projects, and allow tempest to survive being stretched-to-breaking horizontally as we scale to more projects. At the same time, there are benefits to the tempest-as-all-in-one-functional-and-integration-suite that we should try not to lose: 1. Strong integration testing without thinking too hard about the actual dependencies 2. Protection from mistaken or unwise api changes (tempest two-step required) 3. Exportability as a complete blackbox functional test suite that can be used by Rally, RefStack, deployment validation, etc. I think (1) may be the most challenging because tests that are moved out of tempest might be testing some integration that is not being covered by a scenario. We will need to make sure that tempest actually has a complete enough set of tests to validate integration. Even if this is all implemented in a way where tempest can see in-project tests as plugins, there will still not be time to run them all as part of tempest on every commit to every project, so a selection will have to be made. (2) is quite difficult. In Atlanta we talked about taking a copy of functional tests into tempest for stable apis. I don't know how workable that is but don't see any other real options except vigilance in reviews of patches that change functional tests. (3) is what Zane was addressing. The in-project functional tests need to be written in a way that they can, at least in some configuration, run against a real cloud. I suspect from reading the previous thread about In-tree functional test vision that we may actually be dealing with three categories of test here rather than two: * Unit tests that run against the package they are embedded in * Functional tests that run against the package they are embedded in * Integration tests that run against a specified cloud i.e. the tests we are now trying to add to Heat might be qualitatively different from the projectname.tests.functional suites that already exist in a few projects. Perhaps someone from Neutron and/or Swift can confirm? That seems right, except that I would call the third functional tests and not integration tests, because the purpose is not really integration but deep testing of a particular service. Tempest would continue to focus on integration testing. Is there some controversy about that? The second category could include whitebox tests. I don't know about swift, but in neutron the intent was to have these tests be configurable to run against a real cloud, or not. Maru Newby would have details. I'd like to propose that tests of the third type get their own top-level package with a name of the form projectname-integrationtests (second choice: projectname-tempest on the principle that they're essentially plugins for Tempest). How would people feel about standardising that across OpenStack? +1 But I would not call it integrationtests for the reason given above. Because all heat does is interact with other services, what we call functional tests are actually integration tests. Sure, we could mock at the REST API level, but integration coverage is what we need most. This I'd call that faking, not mocking, but both could apply. lets us verify things like: - how heat handles races in other services leading to resources going into ERROR A fake that predictably fails (and thus tests failure handling) will result in better coverage than a real service that only fails when that real service is broken. What's frustrating is that _both_ are needed to catch bugs. - connectivity and interaction between heat and agents on orchestrated servers That is definitely
