Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest
The beginning of this thread largely came from the fact that Marconi clearly doing most of their QA not in an upstream way. To be integrated, that needs to change. Marconi has very good test coverage within the project. These tests guarantee functionality at a project level (i.e the API works as defined in our API docs). But having a good test coverage at the project level, no way implies that we don't want to work with upstream. We want to guarantee quality at every level upstream integration will continue to be one of our major areas of focus. We have never considered the tests within the project and those in Tempest as an 'either or'. We need both - Both of these give valuable feedback , from different perspectives. I've seen this go down with many projects now to the point where it's a normal 5 stages of grief thing. * Denial - we can totally do all this in our own tree, no need to work in Tempest * Anger - what, python*client shipped a new version and we're broken! how dare they? And why do I need to bother working outside my own git tree? * Bargaining - let me propose a whole other approach to testing that doesn't need Tempest * Depression - that's not going to work? why won't you let me gate all the projects on my own thing? ok, fine I'll work with Tempest. * Acceptance - oh, look at that, I just managed to block a Keystone change that would have broken me, but now I have a Tempest test that *they* have to pass as well. Marconi team is not in any of the 'grief' stages ;) We recognize the value of Tempest enhancing test coverage in Tempest is an important goal for us. Is Tempest a paragon of virtue? Far from it. But right now has very clearly shown it's value, and I think remains the right collaboration point to create the contract about what we all believe OpenStack is. We all agree that Tempest is the 'right collaboration point to create the contract about what we all believe OpenStack is'. Making projects more accountable for quality will no way diminish the value of Tempest. On the other hand, Tempest will become more valuable than ever because of the increased focus in integration testing. We need the emphasis on quality to permeate at every level of Openstack. This is a cultural change that needs to take place openstack-qa should be the drivers of this change. Malini ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest
On Wed, Mar 26, 2014 at 9:02 PM, Maru Newby ma...@redhat.com wrote: On Mar 26, 2014, at 12:59 PM, Joe Gordon joe.gord...@gmail.com wrote: On Tue, Mar 25, 2014 at 1:49 AM, Maru Newby ma...@redhat.com wrote: On Mar 21, 2014, at 9:01 AM, David Kranz dkr...@redhat.com wrote: On 03/20/2014 04:19 PM, Rochelle.RochelleGrober wrote: -Original Message- From: Malini Kamalambal [mailto:malini.kamalam...@rackspace.com] Sent: Thursday, March 20, 2014 12:13 PM 'project specific functional testing' in the Marconi context is treating Marconi as a complete system, making Marconi API calls verifying the response - just like an end user would, but without keystone. If one of these tests fail, it is because there is a bug in the Marconi code , and not because its interaction with Keystone caused it to fail. That being said there are certain cases where having a project specific functional test makes sense. For example swift has a functional test job that starts swift in devstack. But, those things are normally handled on a per case basis. In general if the project is meant to be part of the larger OpenStack ecosystem then Tempest is the place to put functional testing. That way you know it works with all of the other components. The thing is in openstack what seems like a project isolated functional test almost always involves another project in real use cases. (for example keystone auth with api requests) One of the concerns we heard in the review was 'having the functional tests elsewhere (I.e within the project itself) does not count and they have to be in Tempest'. This has made us as a team wonder if we should migrate all our functional tests to Tempest. But from Matt's response, I think it is reasonable to continue in our current path have the functional tests in Marconi coexist along with the tests in Tempest. I think that what is being asked, really is that the functional tests could be a single set of tests that would become a part of the tempest repository and that these tests would have an ENV variable as part of the configuration that would allow either no Keystone or Keystone or some such, if that is the only configuration issue that separates running the tests isolated vs. integrated. The functional tests need to be as much as possible a single set of tests to reduce duplication and remove the likelihood of two sets getting out of sync with each other/development. If they only run in the integrated environment, that's ok, but if you want to run them isolated to make debugging easier, then it should be a configuration option and a separate test job. So, if my assumptions are correct, QA only requires functional tests for integrated runs, but if the project QAs/Devs want to run isolated for dev and devtest purposes, more power to them. Just keep it a single set of functional tests and put them in the Tempest repository so that if a failure happens, anyone can find the test and do the debug work without digging into a separate project repository. Hopefully, the tests as designed could easily take a new configuration directive and a short bit of work with OS QA will get the integrated FTs working as well as the isolated ones. --Rocky This issue has been much debated. There are some active members of our community who believe that all the functional tests should live outside of tempest in the projects, albeit with the same idea that such tests could be run either as part of today's real tempest runs or mocked in various ways to allow component isolation or better performance. Maru Newby posted a patch with an example of one way to do this but I think it expired and I don't have a pointer. I think the best place for functional api tests to be maintained is in the projects themselves. The domain expertise required to write api tests is likely to be greater among project resources, and they should be tasked with writing api tests pre-merge. The current 'merge-first, test-later' procedure of maintaining api tests in the Tempest repo makes that impossible. Worse, the cost of developing functional api tests is higher in the integration environment that is the Tempest default. If an API is made and documented properly what domain expertise would be needed to use it? The opposite is true for tempest and the tests themselves. The tempest team focuses on just tests so they know how to write good tests and are able to leverage common underlying framework code. Given that documentation is typically finalized only late in the cycle, are you suggesting that we forego api testing until possibly well after the code has been written? Plus, it is a rare api that doesn't have to evolve in response to real-world experience with early implementations. The sooner we can write
Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest
On Wed, 26 Mar 2014 21:02:58 -0700 Maru Newby ma...@redhat.com wrote: If an API is made and documented properly what domain expertise would be needed to use it? The opposite is true for tempest and the tests themselves. The tempest team focuses on just tests so they know how to write good tests and are able to leverage common underlying framework code. Given that documentation is typically finalized only late in the cycle, are you suggesting that we forego api testing until possibly well after the code has been written? Plus, it is a rare api that doesn't have to evolve in response to real-world experience with early implementations. The sooner we can write functional api tests, the sooner we can identify shortcomings that need to be addressed - and the less costly they will be to fix. So although proper documentation may only be finalized late in the cycle, there really should be a specification of the API and how it behaves done before the implementation is finished. At least in Nova land I think the lack of this has been a major cause of features having trouble merging and also us having flaws in both our implementation (semantic behaviour which was accidental rather than designed) and testing (incomplete coverage). Also we have API unit testing in Nova but the tempest API testing still ends up picking up more issues (perhaps because its generally not the person writing the code who ends up writing the tests). I think it also increases the chance that a backwards incompatible API changes will get picked up. Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest
This is really interesting discussion but was thrown off by the different use of ‘functional testing.’ I decided to reconcile it with my understanding and ended up with this two-pager. Sharing it in case it helps: https://docs.google.com/document/d/1ILxfoJlov9lBfuuZtvwW_7bmlGaYYw0UsE1R9lo6XwQ/edit Regards, Mark Maglana QA/Release Engineer Nexus IS, Inc. Single Number Reach: 424-225-1309 On Mar 25, 2014, at 4:55 AM, Malini Kamalambal malini.kamalam...@rackspace.com wrote: We are talking about different levels of testing, 1. Unit tests - which everybody agrees should be in the individual project itself 2. System Tests - 'System' referring to ( limited to), all the components that make up the project. These are also the functional tests for the project. 3. Integration Tests - This is to verify that the OS components interact well and don't break other components -Keystone being the most obvious example. This is where I see getting the maximum mileage out of Tempest. Its not easy to detect what the integration points with other projects are, any project can use any stable API from any other project. Because of this all OpenStack APIs should fit into this category. Any project can use any stable API –but that does not make all API tests , Integration Tests. A test becomes Integration test when it has two or more projects interacting in the test. Individual projects should be held accountable to make sure that their API's work – no matter who consumes them. We should be able to treat the project as a complete system, make API calls and validate that the response matches the API definitions. Identifying issues earlier in the pipeline reduces the Total Cost of Quality. I agree that Integration Testing is hard. It is complicated because it requires knowledge of how systems interact with each other – and knowledge comes from a lot of time spent on analysis. It requires people with project expertise to talk to each other identify possible test cases. openstack-qa is the ideal forum to do this. Holding projects responsible for their functionality will help the QA team focus on complicated integration tests. Having a second group writing tests to Nova's public APIs has been really helpful in keeping us honest as well. Sounds like a testimonial for more project level testing :) I see value in projects taking ownership of the System Tests - because if the project is not 'functionally ready', it is not ready to integrate with other components of Openstack. What do you mean by not ready? 'Functionally Ready' - The units that make up a project can work together as a system, all API's have been exercised with positive negative test cases by treating the project as a complete system. There are no known critical bugs. The point here being identify as many issues as possible, earlier in the game. But for this approach to be successful, projects should have diversity in the team composition - we need more testers who focus on creating these tests. This will keep the teams honest in their quality standards. As long as individual projects cannot guarantee functional test coverage, we will need more tests in Tempest. But that will shift focus away from Integration Testing, which can be done ONLY in Tempest. Regardless of whatever we end up deciding, it will be good to have these discussions sooner than later. This will help at least the new projects to move in the right direction. -Malini ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest
On Tue, Mar 25, 2014 at 1:49 AM, Maru Newby ma...@redhat.com wrote: On Mar 21, 2014, at 9:01 AM, David Kranz dkr...@redhat.com wrote: On 03/20/2014 04:19 PM, Rochelle.RochelleGrober wrote: -Original Message- From: Malini Kamalambal [mailto:malini.kamalam...@rackspace.com] Sent: Thursday, March 20, 2014 12:13 PM 'project specific functional testing' in the Marconi context is treating Marconi as a complete system, making Marconi API calls verifying the response - just like an end user would, but without keystone. If one of these tests fail, it is because there is a bug in the Marconi code , and not because its interaction with Keystone caused it to fail. That being said there are certain cases where having a project specific functional test makes sense. For example swift has a functional test job that starts swift in devstack. But, those things are normally handled on a per case basis. In general if the project is meant to be part of the larger OpenStack ecosystem then Tempest is the place to put functional testing. That way you know it works with all of the other components. The thing is in openstack what seems like a project isolated functional test almost always involves another project in real use cases. (for example keystone auth with api requests) One of the concerns we heard in the review was 'having the functional tests elsewhere (I.e within the project itself) does not count and they have to be in Tempest'. This has made us as a team wonder if we should migrate all our functional tests to Tempest. But from Matt's response, I think it is reasonable to continue in our current path have the functional tests in Marconi coexist along with the tests in Tempest. I think that what is being asked, really is that the functional tests could be a single set of tests that would become a part of the tempest repository and that these tests would have an ENV variable as part of the configuration that would allow either no Keystone or Keystone or some such, if that is the only configuration issue that separates running the tests isolated vs. integrated. The functional tests need to be as much as possible a single set of tests to reduce duplication and remove the likelihood of two sets getting out of sync with each other/development. If they only run in the integrated environment, that's ok, but if you want to run them isolated to make debugging easier, then it should be a configuration option and a separate test job. So, if my assumptions are correct, QA only requires functional tests for integrated runs, but if the project QAs/Devs want to run isolated for dev and devtest purposes, more power to them. Just keep it a single set of functional tests and put them in the Tempest repository so that if a failure happens, anyone can find the test and do the debug work without digging into a separate project repository. Hopefully, the tests as designed could easily take a new configuration directive and a short bit of work with OS QA will get the integrated FTs working as well as the isolated ones. --Rocky This issue has been much debated. There are some active members of our community who believe that all the functional tests should live outside of tempest in the projects, albeit with the same idea that such tests could be run either as part of today's real tempest runs or mocked in various ways to allow component isolation or better performance. Maru Newby posted a patch with an example of one way to do this but I think it expired and I don't have a pointer. I think the best place for functional api tests to be maintained is in the projects themselves. The domain expertise required to write api tests is likely to be greater among project resources, and they should be tasked with writing api tests pre-merge. The current 'merge-first, test-later' procedure of maintaining api tests in the Tempest repo makes that impossible. Worse, the cost of developing functional api tests is higher in the integration environment that is the Tempest default. If an API is made and documented properly what domain expertise would be needed to use it? The opposite is true for tempest and the tests themselves. The tempest team focuses on just tests so they know how to write good tests and are able to leverage common underlying framework code. Yes 'merge-first, test-later' is bad. But There are other ways of fixing this then moving the tests out of tempest. Hopefully cross project dependencies in zuul will help make this workflow easier. The patch in question [1] proposes allowing pre-merge functional api test maintenance and test reuse in an integration environment. m. 1: https://review.openstack.org/#/c/72585/ IMO there are valid arguments on both sides, but I hope every one could agree that functional tests should not be arbitrarily split between projects and
Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest
On 03/25/2014 04:49 AM, Maru Newby wrote: On Mar 21, 2014, at 9:01 AM, David Kranz dkr...@redhat.com wrote: On 03/20/2014 04:19 PM, Rochelle.RochelleGrober wrote: -Original Message- From: Malini Kamalambal [mailto:malini.kamalam...@rackspace.com] Sent: Thursday, March 20, 2014 12:13 PM 'project specific functional testing' in the Marconi context is treating Marconi as a complete system, making Marconi API calls verifying the response - just like an end user would, but without keystone. If one of these tests fail, it is because there is a bug in the Marconi code , and not because its interaction with Keystone caused it to fail. That being said there are certain cases where having a project specific functional test makes sense. For example swift has a functional test job that starts swift in devstack. But, those things are normally handled on a per case basis. In general if the project is meant to be part of the larger OpenStack ecosystem then Tempest is the place to put functional testing. That way you know it works with all of the other components. The thing is in openstack what seems like a project isolated functional test almost always involves another project in real use cases. (for example keystone auth with api requests) One of the concerns we heard in the review was 'having the functional tests elsewhere (I.e within the project itself) does not count and they have to be in Tempest'. This has made us as a team wonder if we should migrate all our functional tests to Tempest. But from Matt's response, I think it is reasonable to continue in our current path have the functional tests in Marconi coexist along with the tests in Tempest. I think that what is being asked, really is that the functional tests could be a single set of tests that would become a part of the tempest repository and that these tests would have an ENV variable as part of the configuration that would allow either no Keystone or Keystone or some such, if that is the only configuration issue that separates running the tests isolated vs. integrated. The functional tests need to be as much as possible a single set of tests to reduce duplication and remove the likelihood of two sets getting out of sync with each other/development. If they only run in the integrated environment, that's ok, but if you want to run them isolated to make debugging easier, then it should be a configuration option and a separate test job. So, if my assumptions are correct, QA only requires functional tests for integrated runs, but if the project QAs/Devs want to run isolated for dev and devtest purposes, more power to them. Just keep it a single set of functional tests and put them in the Tempest repository so that if a failure happens, anyone can find the test and do the debug work without digging into a separate project repository. Hopefully, the tests as designed could easily take a new configuration directive and a short bit of work with OS QA will get the integrated FTs working as well as the isolated ones. --Rocky This issue has been much debated. There are some active members of our community who believe that all the functional tests should live outside of tempest in the projects, albeit with the same idea that such tests could be run either as part of today's real tempest runs or mocked in various ways to allow component isolation or better performance. Maru Newby posted a patch with an example of one way to do this but I think it expired and I don't have a pointer. I think the best place for functional api tests to be maintained is in the projects themselves. The domain expertise required to write api tests is likely to be greater among project resources, and they should be tasked with writing api tests pre-merge. The current 'merge-first, test-later' procedure of maintaining api tests in the Tempest repo makes that impossible. Worse, the cost of developing functional api tests is higher in the integration environment that is the Tempest default. I disagree. I think all that ends up doing is creating greater variance in quality between components in OpenStack. And it means now no one feels responsible when a bad test in a project some where causes a gate block. If a core project team can't be bothered to work in the docs, infra, and qa, and other project repositories repos as part normal flow, that's core project is very clearly not integrated with the rest of OpenStack. Being integrated needs to not just be a badge you get from the TC on a vote, it actually means being integrated, beyond just your own git tree that you have +2 on. The patch in question [1] proposes allowing pre-merge functional api test maintenance and test reuse in an integration environment. m. 1: https://review.openstack.org/#/c/72585/ This effort is interesting, but I feel like it's so far down on maslo's hierarchy of needs,
Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest
On Mar 26, 2014, at 12:59 PM, Joe Gordon joe.gord...@gmail.com wrote: On Tue, Mar 25, 2014 at 1:49 AM, Maru Newby ma...@redhat.com wrote: On Mar 21, 2014, at 9:01 AM, David Kranz dkr...@redhat.com wrote: On 03/20/2014 04:19 PM, Rochelle.RochelleGrober wrote: -Original Message- From: Malini Kamalambal [mailto:malini.kamalam...@rackspace.com] Sent: Thursday, March 20, 2014 12:13 PM 'project specific functional testing' in the Marconi context is treating Marconi as a complete system, making Marconi API calls verifying the response - just like an end user would, but without keystone. If one of these tests fail, it is because there is a bug in the Marconi code , and not because its interaction with Keystone caused it to fail. That being said there are certain cases where having a project specific functional test makes sense. For example swift has a functional test job that starts swift in devstack. But, those things are normally handled on a per case basis. In general if the project is meant to be part of the larger OpenStack ecosystem then Tempest is the place to put functional testing. That way you know it works with all of the other components. The thing is in openstack what seems like a project isolated functional test almost always involves another project in real use cases. (for example keystone auth with api requests) One of the concerns we heard in the review was 'having the functional tests elsewhere (I.e within the project itself) does not count and they have to be in Tempest'. This has made us as a team wonder if we should migrate all our functional tests to Tempest. But from Matt's response, I think it is reasonable to continue in our current path have the functional tests in Marconi coexist along with the tests in Tempest. I think that what is being asked, really is that the functional tests could be a single set of tests that would become a part of the tempest repository and that these tests would have an ENV variable as part of the configuration that would allow either no Keystone or Keystone or some such, if that is the only configuration issue that separates running the tests isolated vs. integrated. The functional tests need to be as much as possible a single set of tests to reduce duplication and remove the likelihood of two sets getting out of sync with each other/development. If they only run in the integrated environment, that's ok, but if you want to run them isolated to make debugging easier, then it should be a configuration option and a separate test job. So, if my assumptions are correct, QA only requires functional tests for integrated runs, but if the project QAs/Devs want to run isolated for dev and devtest purposes, more power to them. Just keep it a single set of functional tests and put them in the Tempest repository so that if a failure happens, anyone can find the test and do the debug work without digging into a separate project repository. Hopefully, the tests as designed could easily take a new configuration directive and a short bit of work with OS QA will get the integrated FTs working as well as the isolated ones. --Rocky This issue has been much debated. There are some active members of our community who believe that all the functional tests should live outside of tempest in the projects, albeit with the same idea that such tests could be run either as part of today's real tempest runs or mocked in various ways to allow component isolation or better performance. Maru Newby posted a patch with an example of one way to do this but I think it expired and I don't have a pointer. I think the best place for functional api tests to be maintained is in the projects themselves. The domain expertise required to write api tests is likely to be greater among project resources, and they should be tasked with writing api tests pre-merge. The current 'merge-first, test-later' procedure of maintaining api tests in the Tempest repo makes that impossible. Worse, the cost of developing functional api tests is higher in the integration environment that is the Tempest default. If an API is made and documented properly what domain expertise would be needed to use it? The opposite is true for tempest and the tests themselves. The tempest team focuses on just tests so they know how to write good tests and are able to leverage common underlying framework code. Given that documentation is typically finalized only late in the cycle, are you suggesting that we forego api testing until possibly well after the code has been written? Plus, it is a rare api that doesn't have to evolve in response to real-world experience with early implementations. The sooner we can write functional api tests, the sooner we can identify shortcomings that need to be addressed
Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest
On Mar 26, 2014, at 1:52 PM, Sean Dague s...@dague.net wrote: On 03/25/2014 04:49 AM, Maru Newby wrote: On Mar 21, 2014, at 9:01 AM, David Kranz dkr...@redhat.com wrote: On 03/20/2014 04:19 PM, Rochelle.RochelleGrober wrote: -Original Message- From: Malini Kamalambal [mailto:malini.kamalam...@rackspace.com] Sent: Thursday, March 20, 2014 12:13 PM 'project specific functional testing' in the Marconi context is treating Marconi as a complete system, making Marconi API calls verifying the response - just like an end user would, but without keystone. If one of these tests fail, it is because there is a bug in the Marconi code , and not because its interaction with Keystone caused it to fail. That being said there are certain cases where having a project specific functional test makes sense. For example swift has a functional test job that starts swift in devstack. But, those things are normally handled on a per case basis. In general if the project is meant to be part of the larger OpenStack ecosystem then Tempest is the place to put functional testing. That way you know it works with all of the other components. The thing is in openstack what seems like a project isolated functional test almost always involves another project in real use cases. (for example keystone auth with api requests) One of the concerns we heard in the review was 'having the functional tests elsewhere (I.e within the project itself) does not count and they have to be in Tempest'. This has made us as a team wonder if we should migrate all our functional tests to Tempest. But from Matt's response, I think it is reasonable to continue in our current path have the functional tests in Marconi coexist along with the tests in Tempest. I think that what is being asked, really is that the functional tests could be a single set of tests that would become a part of the tempest repository and that these tests would have an ENV variable as part of the configuration that would allow either no Keystone or Keystone or some such, if that is the only configuration issue that separates running the tests isolated vs. integrated. The functional tests need to be as much as possible a single set of tests to reduce duplication and remove the likelihood of two sets getting out of sync with each other/development. If they only run in the integrated environment, that's ok, but if you want to run them isolated to make debugging easier, then it should be a configuration option and a separate test job. So, if my assumptions are correct, QA only requires functional tests for integrated runs, but if the project QAs/Devs want to run isolated for dev and devtest purposes, more power to them. Just keep it a single set of functional tests and put them in the Tempest repository so that if a failure happens, anyone can find the test and do the debug work without digging into a separate project repository. Hopefully, the tests as designed could easily take a new configuration directive and a short bit of work with OS QA will get the integrated FTs working as well as the isolated ones. --Rocky This issue has been much debated. There are some active members of our community who believe that all the functional tests should live outside of tempest in the projects, albeit with the same idea that such tests could be run either as part of today's real tempest runs or mocked in various ways to allow component isolation or better performance. Maru Newby posted a patch with an example of one way to do this but I think it expired and I don't have a pointer. I think the best place for functional api tests to be maintained is in the projects themselves. The domain expertise required to write api tests is likely to be greater among project resources, and they should be tasked with writing api tests pre-merge. The current 'merge-first, test-later' procedure of maintaining api tests in the Tempest repo makes that impossible. Worse, the cost of developing functional api tests is higher in the integration environment that is the Tempest default. I disagree. I think all that ends up doing is creating greater variance in quality between components in OpenStack. And it means now no one feels responsible when a bad test in a project some where causes a gate block. In my experience, projects have greater incentive to maintain tests that are in the same tree as the code they are testing. The tests can run pre-merge, and the same eyes that review the code can review the tests for that code. Any lag between when code is written and tests are written, at least below the integration level, can be detrimental due to the fact that the mental models for the code naturally decay over time. Does your experience on this differ? As you say, though, maybe its worth taking the hit of not putting as much testing in-tree as possible. Maybe
Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest
On Mar 21, 2014, at 9:01 AM, David Kranz dkr...@redhat.com wrote: On 03/20/2014 04:19 PM, Rochelle.RochelleGrober wrote: -Original Message- From: Malini Kamalambal [mailto:malini.kamalam...@rackspace.com] Sent: Thursday, March 20, 2014 12:13 PM 'project specific functional testing' in the Marconi context is treating Marconi as a complete system, making Marconi API calls verifying the response - just like an end user would, but without keystone. If one of these tests fail, it is because there is a bug in the Marconi code , and not because its interaction with Keystone caused it to fail. That being said there are certain cases where having a project specific functional test makes sense. For example swift has a functional test job that starts swift in devstack. But, those things are normally handled on a per case basis. In general if the project is meant to be part of the larger OpenStack ecosystem then Tempest is the place to put functional testing. That way you know it works with all of the other components. The thing is in openstack what seems like a project isolated functional test almost always involves another project in real use cases. (for example keystone auth with api requests) One of the concerns we heard in the review was 'having the functional tests elsewhere (I.e within the project itself) does not count and they have to be in Tempest'. This has made us as a team wonder if we should migrate all our functional tests to Tempest. But from Matt's response, I think it is reasonable to continue in our current path have the functional tests in Marconi coexist along with the tests in Tempest. I think that what is being asked, really is that the functional tests could be a single set of tests that would become a part of the tempest repository and that these tests would have an ENV variable as part of the configuration that would allow either no Keystone or Keystone or some such, if that is the only configuration issue that separates running the tests isolated vs. integrated. The functional tests need to be as much as possible a single set of tests to reduce duplication and remove the likelihood of two sets getting out of sync with each other/development. If they only run in the integrated environment, that's ok, but if you want to run them isolated to make debugging easier, then it should be a configuration option and a separate test job. So, if my assumptions are correct, QA only requires functional tests for integrated runs, but if the project QAs/Devs want to run isolated for dev and devtest purposes, more power to them. Just keep it a single set of functional tests and put them in the Tempest repository so that if a failure happens, anyone can find the test and do the debug work without digging into a separate project repository. Hopefully, the tests as designed could easily take a new configuration directive and a short bit of work with OS QA will get the integrated FTs working as well as the isolated ones. --Rocky This issue has been much debated. There are some active members of our community who believe that all the functional tests should live outside of tempest in the projects, albeit with the same idea that such tests could be run either as part of today's real tempest runs or mocked in various ways to allow component isolation or better performance. Maru Newby posted a patch with an example of one way to do this but I think it expired and I don't have a pointer. I think the best place for functional api tests to be maintained is in the projects themselves. The domain expertise required to write api tests is likely to be greater among project resources, and they should be tasked with writing api tests pre-merge. The current 'merge-first, test-later' procedure of maintaining api tests in the Tempest repo makes that impossible. Worse, the cost of developing functional api tests is higher in the integration environment that is the Tempest default. The patch in question [1] proposes allowing pre-merge functional api test maintenance and test reuse in an integration environment. m. 1: https://review.openstack.org/#/c/72585/ IMO there are valid arguments on both sides, but I hope every one could agree that functional tests should not be arbitrarily split between projects and tempest as they are now. The Tempest README states a desire for complete coverage of the OpenStack API but Tempest is not close to that. We have been discussing and then ignoring this issue for some time but I think the recent action to say that Tempest will be used to determine if something can use the OpenStack trademark will force more completeness on tempest (more tests, that is). I think we need to resolve this issue but it won't be easy and modifying existing api tests to be more flexible will be a lot of work. But at least new projects could get on the right
Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest
We are talking about different levels of testing, 1. Unit tests - which everybody agrees should be in the individual project itself 2. System Tests - 'System' referring to ( limited to), all the components that make up the project. These are also the functional tests for the project. 3. Integration Tests - This is to verify that the OS components interact well and don't break other components -Keystone being the most obvious example. This is where I see getting the maximum mileage out of Tempest. Its not easy to detect what the integration points with other projects are, any project can use any stable API from any other project. Because of this all OpenStack APIs should fit into this category. Any project can use any stable API –but that does not make all API tests , Integration Tests. A test becomes Integration test when it has two or more projects interacting in the test. Individual projects should be held accountable to make sure that their API's work – no matter who consumes them. We should be able to treat the project as a complete system, make API calls and validate that the response matches the API definitions. Identifying issues earlier in the pipeline reduces the Total Cost of Quality. I agree that Integration Testing is hard. It is complicated because it requires knowledge of how systems interact with each other – and knowledge comes from a lot of time spent on analysis. It requires people with project expertise to talk to each other identify possible test cases. openstack-qa is the ideal forum to do this. Holding projects responsible for their functionality will help the QA team focus on complicated integration tests. Having a second group writing tests to Nova's public APIs has been really helpful in keeping us honest as well. Sounds like a testimonial for more project level testing :) I see value in projects taking ownership of the System Tests - because if the project is not 'functionally ready', it is not ready to integrate with other components of Openstack. What do you mean by not ready? 'Functionally Ready' - The units that make up a project can work together as a system, all API's have been exercised with positive negative test cases by treating the project as a complete system. There are no known critical bugs. The point here being identify as many issues as possible, earlier in the game. But for this approach to be successful, projects should have diversity in the team composition - we need more testers who focus on creating these tests. This will keep the teams honest in their quality standards. As long as individual projects cannot guarantee functional test coverage, we will need more tests in Tempest. But that will shift focus away from Integration Testing, which can be done ONLY in Tempest. Regardless of whatever we end up deciding, it will be good to have these discussions sooner than later. This will help at least the new projects to move in the right direction. -Malini ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest
On 3/21/14 3:49 PM, Rochelle.RochelleGrober rochelle.gro...@huawei.com wrote: From: Malini Kamalambal [mailto:malini.kamalam...@rackspace.com] snip We are talking about different levels of testing, 1. Unit tests - which everybody agrees should be in the individual project itself 2. System Tests - 'System' referring to ( limited to), all the components that make up the project. These are also the functional tests for the project. 3. Integration Tests - This is to verify that the OS components interact well and don't break other components -Keystone being the most obvious example. This is where I see getting the maximum mileage out of Tempest. I see value in projects taking ownership of the System Tests - because if the project is not 'functionally ready', it is not ready to integrate with other components of Openstack. But for this approach to be successful, projects should have diversity in the team composition - we need more testers who focus on creating these tests. This will keep the teams honest in their quality standards. +1000 I love your approach to this. You are right. Functional tests for the project, that exist in an environment, but that exercise the intricacies of just the project aren't there for most projects, but really should be. And these tests should be exercised against new code before the code enters the gerrit/Jenkins stream. But, as Malini points out, it's at most a dream for most projects as the test developers just aren't part of most projects. As long as individual projects cannot guarantee functional test coverage, we will need more tests in Tempest. But that will shift focus away from Integration Testing, which can be done ONLY in Tempest. +1 This is also an important point. If functional testing belonged to the projects, then most of these tests would be run before a tempest test was ever run and would not need to be part of the integrated tests, except as a subset that demonstrate the functioning integration with other projects. Regardless of whatever we end up deciding, it will be good to have these discussions sooner than later. This will help at least the new projects to move in the right direction. Maybe a summit topic? How do we push functional testing into the project level development? --Rocky That is a good idea I just suggested a session for the ATL summit. See http://summit.openstack.org/cfp/details/134 -Malini ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest
On Fri, Mar 21, 2014 at 9:38 AM, Malini Kamalambal malini.kamalam...@rackspace.com wrote: On 3/21/14 12:01 PM, David Kranz dkr...@redhat.com wrote: On 03/20/2014 04:19 PM, Rochelle.RochelleGrober wrote: -Original Message- From: Malini Kamalambal [mailto:malini.kamalam...@rackspace.com] Sent: Thursday, March 20, 2014 12:13 PM 'project specific functional testing' in the Marconi context is treating Marconi as a complete system, making Marconi API calls verifying the response - just like an end user would, but without keystone. If one of these tests fail, it is because there is a bug in the Marconi code , and not because its interaction with Keystone caused it to fail. That being said there are certain cases where having a project specific functional test makes sense. For example swift has a functional test job that starts swift in devstack. But, those things are normally handled on a per case basis. In general if the project is meant to be part of the larger OpenStack ecosystem then Tempest is the place to put functional testing. That way you know it works with all of the other components. The thing is in openstack what seems like a project isolated functional test almost always involves another project in real use cases. (for example keystone auth with api requests) One of the concerns we heard in the review was 'having the functional tests elsewhere (I.e within the project itself) does not count and they have to be in Tempest'. This has made us as a team wonder if we should migrate all our functional tests to Tempest. But from Matt's response, I think it is reasonable to continue in our current path have the functional tests in Marconi coexist along with the tests in Tempest. I think that what is being asked, really is that the functional tests could be a single set of tests that would become a part of the tempest repository and that these tests would have an ENV variable as part of the configuration that would allow either no Keystone or Keystone or some such, if that is the only configuration issue that separates running the tests isolated vs. integrated. The functional tests need to be as much as possible a single set of tests to reduce duplication and remove the likelihood of two sets getting out of sync with each other/development. If they only run in the integrated environment, that's ok, but if you want to run them isolated to make debugging easier, then it should be a configuration option and a separate test job. So, if my assumptions are correct, QA only requires functional tests for integrated runs, but if the project QAs/Devs want to run isolated for dev and devtest purposes, more power to them. Just keep it a single set of functional tests and put them in the Tempest repository so that if a failure happens, anyone can find the test and do the debug work without digging into a separate project repository. Hopefully, the tests as designed could easily take a new configuration directive and a short bit of work with OS QA will get the integrated FTs working as well as the isolated ones. --Rocky This issue has been much debated. There are some active members of our community who believe that all the functional tests should live outside of tempest in the projects, albeit with the same idea that such tests could be run either as part of today's real tempest runs or mocked in various ways to allow component isolation or better performance. Maru Newby posted a patch with an example of one way to do this but I think it expired and I don't have a pointer. IMO there are valid arguments on both sides, but I hope every one could agree that functional tests should not be arbitrarily split between projects and tempest as they are now. The Tempest README states a desire for complete coverage of the OpenStack API but Tempest is not close to that. We have been discussing and then ignoring this issue for some time but I think the recent action to say that Tempest will be used to determine if something can use the OpenStack trademark will force more completeness on tempest (more tests, that is). I think we need to resolve this issue but it won't be easy and modifying existing api tests to be more flexible will be a lot of work. But at least new projects could get on the right path sooner. -David ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev We are talking about different levels of testing, 1. Unit tests - which everybody agrees should be in the individual project itself 2. System Tests - 'System' referring to ( limited to), all the components that make up the project. These are also the functional tests for the project. 3. Integration Tests - This is to verify that the OS components interact well and don't
Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest
I have an etherpad started to document QA requirements https://etherpad.openstack.org/p/Tempest-Graduation-Criteria I hope Sean and the rest of QA team can add their thoughts here. I am also looking for inputs from the Sahara team, while the path to graduation is still fresh in their minds. We can solidify this after the release work is done. Meanwhile the Marconi team gets the message for engaging in active conversations with the upstream community. We look forward to being part of these conversation becoming active contributors in other areas. On 3/20/14 4:17 PM, Sean Dague s...@dague.net wrote: I will agree that the TC language is not as strong as it should be (and really should be clarified, but I don't think that's going to happen until the release is looking solid). Honestly, though, I think Sahara is a good example here of the level of that we expect. They have actively engaged with upstream infra and used it to the full extent possible. Then wrote additional tooling to do even more and report 3rd party in on their changes. It's also worth noting they got there by actively participating in the upstream community and conversations, and clearly making upstream test integration a top priority for the cycle. On 03/20/2014 03:13 PM, Malini Kamalambal wrote: Thanks Matt for your response !! It has clarified some of the 'cloudy areas' ;) So having only looked at the Marconi ML thread and not the actual TC meeting minutes I might be missing the whole picture. But, from what I saw when I looked at both a marconi commit and a tempest commit is that there is no gating marconi devstack-gate job on marconi commits. It's only non-voting in the check pipeline. Additionally, there isn't a non-voting job on tempest or devstack-gate. For example, look at how savanna has it's tempest jobs setup and this is what marconi needs to have. I am not dismissing the fact that Marconi does not have a voting gate job. All we have currently is a non-voting check job in Marconi, and experimental jobs in Tempest devstack-gate. So we definitely have more work to do there, starting with making the devstack-tempest job voting in Marconi. But what caught me by surprise is 'the extend of gate testing - http://git.openstack.org/cgit/openstack/tempest/tree/tempest/api/queuing/ te st_queues.py' was brought up as a concern. We have always had an extensive functional test suite in Marconi , which is run at gate against a marconi-server that the test spins up (This was implemented before we had devstack integration in place. It is in our plans to run the functional test suite in Marconi against devstack Marconi). But my point is not do a postmortem on everything that happened, rather come up with an objective list of items that I (or anybody who wants to meet the Tempest criteria for graduation- No way meant to imply that Tempest work stops after graduation ;) ) need to complete. I started a wiki page to document what will be a good enough graduation criteria to graduate from the QA perspective. https://etherpad.openstack.org/p/Tempest-Graduation-Criteria It will help if the QA team can add their thoughts to this. This can maybe become a recommendation to the TC on what the QA graduation requirements are. What do you mean by project specific functional testing? What makes debugging a marconi failure in a tempest gate job any more involved than debugging a nova or neutron failure? Part of the point of having an integrated gate is saying that the project works well with all the others in OpenStack. IMO that's not just in project functionality but also in community. When there is an issue with a gate job everyone comes together to work on it. For example if you have a keystone patch that breaks a marconi test in check there is open communication about what happened and how to fix it. 'project specific functional testing' in the Marconi context is treating Marconi as a complete system, making Marconi API calls verifying the response - just like an end user would, but without keystone. If one of these tests fail, it is because there is a bug in the Marconi code , and not because its interaction with Keystone caused it to fail. That being said there are certain cases where having a project specific functional test makes sense. For example swift has a functional test job that starts swift in devstack. But, those things are normally handled on a per case basis. In general if the project is meant to be part of the larger OpenStack ecosystem then Tempest is the place to put functional testing. That way you know it works with all of the other components. The thing is in openstack what seems like a project isolated functional test almost always involves another project in real use cases. (for example keystone auth with api requests) One of the concerns we heard in the review was 'having the functional tests elsewhere (I.e within the project
Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest
On 03/20/2014 04:19 PM, Rochelle.RochelleGrober wrote: -Original Message- From: Malini Kamalambal [mailto:malini.kamalam...@rackspace.com] Sent: Thursday, March 20, 2014 12:13 PM 'project specific functional testing' in the Marconi context is treating Marconi as a complete system, making Marconi API calls verifying the response - just like an end user would, but without keystone. If one of these tests fail, it is because there is a bug in the Marconi code , and not because its interaction with Keystone caused it to fail. That being said there are certain cases where having a project specific functional test makes sense. For example swift has a functional test job that starts swift in devstack. But, those things are normally handled on a per case basis. In general if the project is meant to be part of the larger OpenStack ecosystem then Tempest is the place to put functional testing. That way you know it works with all of the other components. The thing is in openstack what seems like a project isolated functional test almost always involves another project in real use cases. (for example keystone auth with api requests) One of the concerns we heard in the review was 'having the functional tests elsewhere (I.e within the project itself) does not count and they have to be in Tempest'. This has made us as a team wonder if we should migrate all our functional tests to Tempest. But from Matt's response, I think it is reasonable to continue in our current path have the functional tests in Marconi coexist along with the tests in Tempest. I think that what is being asked, really is that the functional tests could be a single set of tests that would become a part of the tempest repository and that these tests would have an ENV variable as part of the configuration that would allow either no Keystone or Keystone or some such, if that is the only configuration issue that separates running the tests isolated vs. integrated. The functional tests need to be as much as possible a single set of tests to reduce duplication and remove the likelihood of two sets getting out of sync with each other/development. If they only run in the integrated environment, that's ok, but if you want to run them isolated to make debugging easier, then it should be a configuration option and a separate test job. So, if my assumptions are correct, QA only requires functional tests for integrated runs, but if the project QAs/Devs want to run isolated for dev and devtest purposes, more power to them. Just keep it a single set of functional tests and put them in the Tempest repository so that if a failure happens, anyone can find the test and do the debug work without digging into a separate project repository. Hopefully, the tests as designed could easily take a new configuration directive and a short bit of work with OS QA will get the integrated FTs working as well as the isolated ones. --Rocky This issue has been much debated. There are some active members of our community who believe that all the functional tests should live outside of tempest in the projects, albeit with the same idea that such tests could be run either as part of today's real tempest runs or mocked in various ways to allow component isolation or better performance. Maru Newby posted a patch with an example of one way to do this but I think it expired and I don't have a pointer. IMO there are valid arguments on both sides, but I hope every one could agree that functional tests should not be arbitrarily split between projects and tempest as they are now. The Tempest README states a desire for complete coverage of the OpenStack API but Tempest is not close to that. We have been discussing and then ignoring this issue for some time but I think the recent action to say that Tempest will be used to determine if something can use the OpenStack trademark will force more completeness on tempest (more tests, that is). I think we need to resolve this issue but it won't be easy and modifying existing api tests to be more flexible will be a lot of work. But at least new projects could get on the right path sooner. -David ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest
On 3/21/14 12:01 PM, David Kranz dkr...@redhat.com wrote: On 03/20/2014 04:19 PM, Rochelle.RochelleGrober wrote: -Original Message- From: Malini Kamalambal [mailto:malini.kamalam...@rackspace.com] Sent: Thursday, March 20, 2014 12:13 PM 'project specific functional testing' in the Marconi context is treating Marconi as a complete system, making Marconi API calls verifying the response - just like an end user would, but without keystone. If one of these tests fail, it is because there is a bug in the Marconi code , and not because its interaction with Keystone caused it to fail. That being said there are certain cases where having a project specific functional test makes sense. For example swift has a functional test job that starts swift in devstack. But, those things are normally handled on a per case basis. In general if the project is meant to be part of the larger OpenStack ecosystem then Tempest is the place to put functional testing. That way you know it works with all of the other components. The thing is in openstack what seems like a project isolated functional test almost always involves another project in real use cases. (for example keystone auth with api requests) One of the concerns we heard in the review was 'having the functional tests elsewhere (I.e within the project itself) does not count and they have to be in Tempest'. This has made us as a team wonder if we should migrate all our functional tests to Tempest. But from Matt's response, I think it is reasonable to continue in our current path have the functional tests in Marconi coexist along with the tests in Tempest. I think that what is being asked, really is that the functional tests could be a single set of tests that would become a part of the tempest repository and that these tests would have an ENV variable as part of the configuration that would allow either no Keystone or Keystone or some such, if that is the only configuration issue that separates running the tests isolated vs. integrated. The functional tests need to be as much as possible a single set of tests to reduce duplication and remove the likelihood of two sets getting out of sync with each other/development. If they only run in the integrated environment, that's ok, but if you want to run them isolated to make debugging easier, then it should be a configuration option and a separate test job. So, if my assumptions are correct, QA only requires functional tests for integrated runs, but if the project QAs/Devs want to run isolated for dev and devtest purposes, more power to them. Just keep it a single set of functional tests and put them in the Tempest repository so that if a failure happens, anyone can find the test and do the debug work without digging into a separate project repository. Hopefully, the tests as designed could easily take a new configuration directive and a short bit of work with OS QA will get the integrated FTs working as well as the isolated ones. --Rocky This issue has been much debated. There are some active members of our community who believe that all the functional tests should live outside of tempest in the projects, albeit with the same idea that such tests could be run either as part of today's real tempest runs or mocked in various ways to allow component isolation or better performance. Maru Newby posted a patch with an example of one way to do this but I think it expired and I don't have a pointer. IMO there are valid arguments on both sides, but I hope every one could agree that functional tests should not be arbitrarily split between projects and tempest as they are now. The Tempest README states a desire for complete coverage of the OpenStack API but Tempest is not close to that. We have been discussing and then ignoring this issue for some time but I think the recent action to say that Tempest will be used to determine if something can use the OpenStack trademark will force more completeness on tempest (more tests, that is). I think we need to resolve this issue but it won't be easy and modifying existing api tests to be more flexible will be a lot of work. But at least new projects could get on the right path sooner. -David ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev We are talking about different levels of testing, 1. Unit tests - which everybody agrees should be in the individual project itself 2. System Tests - 'System' referring to ( limited to), all the components that make up the project. These are also the functional tests for the project. 3. Integration Tests - This is to verify that the OS components interact well and don't break other components -Keystone being the most obvious example. This is where I see getting the maximum mileage out of Tempest. I see value in projects taking ownership of the System Tests -
Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest
From: Malini Kamalambal [mailto:malini.kamalam...@rackspace.com] snip We are talking about different levels of testing, 1. Unit tests - which everybody agrees should be in the individual project itself 2. System Tests - 'System' referring to ( limited to), all the components that make up the project. These are also the functional tests for the project. 3. Integration Tests - This is to verify that the OS components interact well and don't break other components -Keystone being the most obvious example. This is where I see getting the maximum mileage out of Tempest. I see value in projects taking ownership of the System Tests - because if the project is not 'functionally ready', it is not ready to integrate with other components of Openstack. But for this approach to be successful, projects should have diversity in the team composition - we need more testers who focus on creating these tests. This will keep the teams honest in their quality standards. +1000 I love your approach to this. You are right. Functional tests for the project, that exist in an environment, but that exercise the intricacies of just the project aren't there for most projects, but really should be. And these tests should be exercised against new code before the code enters the gerrit/Jenkins stream. But, as Malini points out, it's at most a dream for most projects as the test developers just aren't part of most projects. As long as individual projects cannot guarantee functional test coverage, we will need more tests in Tempest. But that will shift focus away from Integration Testing, which can be done ONLY in Tempest. +1 This is also an important point. If functional testing belonged to the projects, then most of these tests would be run before a tempest test was ever run and would not need to be part of the integrated tests, except as a subset that demonstrate the functioning integration with other projects. Regardless of whatever we end up deciding, it will be good to have these discussions sooner than later. This will help at least the new projects to move in the right direction. Maybe a summit topic? How do we push functional testing into the project level development? --Rocky -Malini ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest
Hello all, I have been working on adding tests in Tempest for Marconi, for the last few months. While there are many amazing people to work with, the process has been more difficult than I expected. Couple of pain-points and suggestions to make the process easier for myself future contributors. 1. The QA requirements for a project to graduate needs details beyond the Project must have a *basic* devstack-gate job set up 2. The scope of Tempest needs clarification - what tests should be in Tempest vs. in the individual projects? Or should they be in both tempest and the project? See details below. 1. There is little documentation on graduation requirement from a QA perspective beyond 'Project must have a basic devstack-gate job set up'. As a result, I hear different interpretations on what a basic devstack gate job is. This topic was discussed in one of the QA meetings a few weeks back [1]. Based on the discussion there, having a basic job - such as one that will let us know 'if a keystone change broke marconi' was good enough. My efforts in getting Marconi meet graduation requirements w.r.t Tempest was based on the above discussion. However, my conversations with the TC during Marconi's graduation review lead me to believe that these requirements aren't yet formalized. We were told that we needed to have more test coverage in tempest, having them elsewhere (i.e. functional tests in the Marconi project itself) was not good enough. I will never debate the value of having good test coverage - after all I define myself professionally as a QA ;) I am proud of the unit and functional test suites the test coverage we have in Marconi [2]. Marconi team is continuing its efforts in this direction. We are looking forward to adding more tests in Tempest and making sure Marconi is in par with the community standards. But what frustrates me is that the test requirements seem to evolve, catching new contributors by surprise. It will really help to have these requirements documented in detail - answering at least the following questions, a. What tests are needed to graduate - API, Scenario, CLI? b. How much coverage is good enough to graduate? That will make sure that contributors focus their time energy in the right direction. I am willing to lead the effort to document the QA-level graduation requirements for a project and help solidify them. 2. Clarify the scope of Tempest - what tests should be in Tempest vs in the individual projects ? It sounds like the scope of tempest is to make sure that, a. Projects are functionally tested (AND) b. Openstack components (a.k.a projects) do not have integration issues. Assuming my understanding is correct, does it make sense to have the project specific functional tests in Tempest? Troubleshooting failures related to project specific functionality requires deep understanding of the individual projects. Isn't it better to leave it to the individual projects to make sure that they are functional? That will help the contributors to Tempest spend their time on what only Tempest can do -i.e. identify integration issues. My hope is that we can clearly outline these requirements as soon as possible. Looking forward to hearing your thoughts!! Best Regards, Malini [1] http://eavesdrop.openstack.org/meetings/qa/2014/qa.2014-02-13-17.00.log.html (See discussion starting with timestamp 17:51:57) [2] https://github.com/openstack/marconi/tree/master/tests ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest
On Thu, Mar 20, 2014 at 11:35:15AM +, Malini Kamalambal wrote: Hello all, I have been working on adding tests in Tempest for Marconi, for the last few months. While there are many amazing people to work with, the process has been more difficult than I expected. Couple of pain-points and suggestions to make the process easier for myself future contributors. 1. The QA requirements for a project to graduate needs details beyond the Project must have a *basic* devstack-gate job set up 2. The scope of Tempest needs clarification - what tests should be in Tempest vs. in the individual projects? Or should they be in both tempest and the project? See details below. 1. There is little documentation on graduation requirement from a QA perspective beyond 'Project must have a basic devstack-gate job set up'. As a result, I hear different interpretations on what a basic devstack gate job is. This topic was discussed in one of the QA meetings a few weeks back [1]. Based on the discussion there, having a basic job - such as one that will let us know 'if a keystone change broke marconi' was good enough. My efforts in getting Marconi meet graduation requirements w.r.t Tempest was based on the above discussion. However, my conversations with the TC during Marconi's graduation review lead me to believe that these requirements aren't yet formalized. We were told that we needed to have more test coverage in tempest, having them elsewhere (i.e. functional tests in the Marconi project itself) was not good enough. So having only looked at the Marconi ML thread and not the actual TC meeting minutes I might be missing the whole picture. But, from what I saw when I looked at both a marconi commit and a tempest commit is that there is no gating marconi devstack-gate job on marconi commits. It's only non-voting in the check pipeline. Additionally, there isn't a non-voting job on tempest or devstack-gate. For example, look at how savanna has it's tempest jobs setup and this is what marconi needs to have. I will never debate the value of having good test coverage - after all I define myself professionally as a QA ;) I am proud of the unit and functional test suites the test coverage we have in Marconi [2]. Marconi team is continuing its efforts in this direction. We are looking forward to adding more tests in Tempest and making sure Marconi is in par with the community standards. But what frustrates me is that the test requirements seem to evolve, catching new contributors by surprise. It will really help to have these requirements documented in detail - answering at least the following questions, a. What tests are needed to graduate - API, Scenario, CLI? b. How much coverage is good enough to graduate? That will make sure that contributors focus their time energy in the right direction. I am willing to lead the effort to document the QA-level graduation requirements for a project and help solidify them. Testing contributions will always be an iterative process. The actual test coverage doesn't matter as much up front. The graduation requirement as I understood it was just to have the glue in place and to verify that everything runs. As long as there is steady contribution and interaction from the marconi community with tempest IMO that matters far more then actually having complete coverage upfront. 2. Clarify the scope of Tempest - what tests should be in Tempest vs in the individual projects ? It sounds like the scope of tempest is to make sure that, a. Projects are functionally tested (AND) b. Openstack components (a.k.a projects) do not have integration issues. Assuming my understanding is correct, does it make sense to have the project specific functional tests in Tempest? Troubleshooting failures related to project specific functionality requires deep understanding of the individual projects. Isn't it better to leave it to the individual projects to make sure that they are functional? That will help the contributors to Tempest spend their time on what only Tempest can do -i.e. identify integration issues. What do you mean by project specific functional testing? What makes debugging a marconi failure in a tempest gate job any more involved than debugging a nova or neutron failure? Part of the point of having an integrated gate is saying that the project works well with all the others in OpenStack. IMO that's not just in project functionality but also in community. When there is an issue with a gate job everyone comes together to work on it. For example if you have a keystone patch that breaks a marconi test in check there is open communication about what happened and how to fix it. That being said there are certain cases where having a project specific functional test makes sense. For example swift has a functional test job that starts swift in devstack. But, those things are normally handled on a per
Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest
Thanks Matt for your response !! It has clarified some of the 'cloudy areas' ;) So having only looked at the Marconi ML thread and not the actual TC meeting minutes I might be missing the whole picture. But, from what I saw when I looked at both a marconi commit and a tempest commit is that there is no gating marconi devstack-gate job on marconi commits. It's only non-voting in the check pipeline. Additionally, there isn't a non-voting job on tempest or devstack-gate. For example, look at how savanna has it's tempest jobs setup and this is what marconi needs to have. I am not dismissing the fact that Marconi does not have a voting gate job. All we have currently is a non-voting check job in Marconi, and experimental jobs in Tempest devstack-gate. So we definitely have more work to do there, starting with making the devstack-tempest job voting in Marconi. But what caught me by surprise is 'the extend of gate testing - http://git.openstack.org/cgit/openstack/tempest/tree/tempest/api/queuing/te st_queues.py' was brought up as a concern. We have always had an extensive functional test suite in Marconi , which is run at gate against a marconi-server that the test spins up (This was implemented before we had devstack integration in place. It is in our plans to run the functional test suite in Marconi against devstack Marconi). But my point is not do a postmortem on everything that happened, rather come up with an objective list of items that I (or anybody who wants to meet the Tempest criteria for graduation- No way meant to imply that Tempest work stops after graduation ;) ) need to complete. I started a wiki page to document what will be a good enough graduation criteria to graduate from the QA perspective. https://etherpad.openstack.org/p/Tempest-Graduation-Criteria It will help if the QA team can add their thoughts to this. This can maybe become a recommendation to the TC on what the QA graduation requirements are. What do you mean by project specific functional testing? What makes debugging a marconi failure in a tempest gate job any more involved than debugging a nova or neutron failure? Part of the point of having an integrated gate is saying that the project works well with all the others in OpenStack. IMO that's not just in project functionality but also in community. When there is an issue with a gate job everyone comes together to work on it. For example if you have a keystone patch that breaks a marconi test in check there is open communication about what happened and how to fix it. 'project specific functional testing' in the Marconi context is treating Marconi as a complete system, making Marconi API calls verifying the response - just like an end user would, but without keystone. If one of these tests fail, it is because there is a bug in the Marconi code , and not because its interaction with Keystone caused it to fail. That being said there are certain cases where having a project specific functional test makes sense. For example swift has a functional test job that starts swift in devstack. But, those things are normally handled on a per case basis. In general if the project is meant to be part of the larger OpenStack ecosystem then Tempest is the place to put functional testing. That way you know it works with all of the other components. The thing is in openstack what seems like a project isolated functional test almost always involves another project in real use cases. (for example keystone auth with api requests) One of the concerns we heard in the review was 'having the functional tests elsewhere (I.e within the project itself) does not count and they have to be in Tempest'. This has made us as a team wonder if we should migrate all our functional tests to Tempest. But from Matt's response, I think it is reasonable to continue in our current path have the functional tests in Marconi coexist along with the tests in Tempest. On 3/20/14 1:59 PM, Matthew Treinish mtrein...@kortar.org wrote: On Thu, Mar 20, 2014 at 11:35:15AM +, Malini Kamalambal wrote: Hello all, I have been working on adding tests in Tempest for Marconi, for the last few months. While there are many amazing people to work with, the process has been more difficult than I expected. Couple of pain-points and suggestions to make the process easier for myself future contributors. 1. The QA requirements for a project to graduate needs details beyond the Project must have a *basic* devstack-gate job set up 2. The scope of Tempest needs clarification - what tests should be in Tempest vs. in the individual projects? Or should they be in both tempest and the project? See details below. 1. There is little documentation on graduation requirement from a QA perspective beyond 'Project must have a basic devstack-gate job set up'. As a result, I hear different interpretations on what a basic devstack gate job is. This topic was discussed in one of the QA meetings a few weeks back [1].
Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest
I will agree that the TC language is not as strong as it should be (and really should be clarified, but I don't think that's going to happen until the release is looking solid). Honestly, though, I think Sahara is a good example here of the level of that we expect. They have actively engaged with upstream infra and used it to the full extent possible. Then wrote additional tooling to do even more and report 3rd party in on their changes. It's also worth noting they got there by actively participating in the upstream community and conversations, and clearly making upstream test integration a top priority for the cycle. On 03/20/2014 03:13 PM, Malini Kamalambal wrote: Thanks Matt for your response !! It has clarified some of the 'cloudy areas' ;) So having only looked at the Marconi ML thread and not the actual TC meeting minutes I might be missing the whole picture. But, from what I saw when I looked at both a marconi commit and a tempest commit is that there is no gating marconi devstack-gate job on marconi commits. It's only non-voting in the check pipeline. Additionally, there isn't a non-voting job on tempest or devstack-gate. For example, look at how savanna has it's tempest jobs setup and this is what marconi needs to have. I am not dismissing the fact that Marconi does not have a voting gate job. All we have currently is a non-voting check job in Marconi, and experimental jobs in Tempest devstack-gate. So we definitely have more work to do there, starting with making the devstack-tempest job voting in Marconi. But what caught me by surprise is 'the extend of gate testing - http://git.openstack.org/cgit/openstack/tempest/tree/tempest/api/queuing/te st_queues.py' was brought up as a concern. We have always had an extensive functional test suite in Marconi , which is run at gate against a marconi-server that the test spins up (This was implemented before we had devstack integration in place. It is in our plans to run the functional test suite in Marconi against devstack Marconi). But my point is not do a postmortem on everything that happened, rather come up with an objective list of items that I (or anybody who wants to meet the Tempest criteria for graduation- No way meant to imply that Tempest work stops after graduation ;) ) need to complete. I started a wiki page to document what will be a good enough graduation criteria to graduate from the QA perspective. https://etherpad.openstack.org/p/Tempest-Graduation-Criteria It will help if the QA team can add their thoughts to this. This can maybe become a recommendation to the TC on what the QA graduation requirements are. What do you mean by project specific functional testing? What makes debugging a marconi failure in a tempest gate job any more involved than debugging a nova or neutron failure? Part of the point of having an integrated gate is saying that the project works well with all the others in OpenStack. IMO that's not just in project functionality but also in community. When there is an issue with a gate job everyone comes together to work on it. For example if you have a keystone patch that breaks a marconi test in check there is open communication about what happened and how to fix it. 'project specific functional testing' in the Marconi context is treating Marconi as a complete system, making Marconi API calls verifying the response - just like an end user would, but without keystone. If one of these tests fail, it is because there is a bug in the Marconi code , and not because its interaction with Keystone caused it to fail. That being said there are certain cases where having a project specific functional test makes sense. For example swift has a functional test job that starts swift in devstack. But, those things are normally handled on a per case basis. In general if the project is meant to be part of the larger OpenStack ecosystem then Tempest is the place to put functional testing. That way you know it works with all of the other components. The thing is in openstack what seems like a project isolated functional test almost always involves another project in real use cases. (for example keystone auth with api requests) One of the concerns we heard in the review was 'having the functional tests elsewhere (I.e within the project itself) does not count and they have to be in Tempest'. This has made us as a team wonder if we should migrate all our functional tests to Tempest. But from Matt's response, I think it is reasonable to continue in our current path have the functional tests in Marconi coexist along with the tests in Tempest. On 3/20/14 1:59 PM, Matthew Treinish mtrein...@kortar.org wrote: On Thu, Mar 20, 2014 at 11:35:15AM +, Malini Kamalambal wrote: Hello all, I have been working on adding tests in Tempest for Marconi, for the last few months. While there are many amazing people to work with, the
Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest
-Original Message- From: Malini Kamalambal [mailto:malini.kamalam...@rackspace.com] Sent: Thursday, March 20, 2014 12:13 PM 'project specific functional testing' in the Marconi context is treating Marconi as a complete system, making Marconi API calls verifying the response - just like an end user would, but without keystone. If one of these tests fail, it is because there is a bug in the Marconi code , and not because its interaction with Keystone caused it to fail. That being said there are certain cases where having a project specific functional test makes sense. For example swift has a functional test job that starts swift in devstack. But, those things are normally handled on a per case basis. In general if the project is meant to be part of the larger OpenStack ecosystem then Tempest is the place to put functional testing. That way you know it works with all of the other components. The thing is in openstack what seems like a project isolated functional test almost always involves another project in real use cases. (for example keystone auth with api requests) One of the concerns we heard in the review was 'having the functional tests elsewhere (I.e within the project itself) does not count and they have to be in Tempest'. This has made us as a team wonder if we should migrate all our functional tests to Tempest. But from Matt's response, I think it is reasonable to continue in our current path have the functional tests in Marconi coexist along with the tests in Tempest. I think that what is being asked, really is that the functional tests could be a single set of tests that would become a part of the tempest repository and that these tests would have an ENV variable as part of the configuration that would allow either no Keystone or Keystone or some such, if that is the only configuration issue that separates running the tests isolated vs. integrated. The functional tests need to be as much as possible a single set of tests to reduce duplication and remove the likelihood of two sets getting out of sync with each other/development. If they only run in the integrated environment, that's ok, but if you want to run them isolated to make debugging easier, then it should be a configuration option and a separate test job. So, if my assumptions are correct, QA only requires functional tests for integrated runs, but if the project QAs/Devs want to run isolated for dev and devtest purposes, more power to them. Just keep it a single set of functional tests and put them in the Tempest repository so that if a failure happens, anyone can find the test and do the debug work without digging into a separate project repository. Hopefully, the tests as designed could easily take a new configuration directive and a short bit of work with OS QA will get the integrated FTs working as well as the isolated ones. --Rocky On 3/20/14 1:59 PM, Matthew Treinish mtrein...@kortar.org wrote: On Thu, Mar 20, 2014 at 11:35:15AM +, Malini Kamalambal wrote: Hello all, I have been working on adding tests in Tempest for Marconi, for the last few months. While there are many amazing people to work with, the process has been more difficult than I expected. Couple of pain-points and suggestions to make the process easier for myself future contributors. 1. The QA requirements for a project to graduate needs details beyond the Project must have a *basic* devstack-gate job set up 2. The scope of Tempest needs clarification - what tests should be in Tempest vs. in the individual projects? Or should they be in both tempest and the project? See details below. 1. There is little documentation on graduation requirement from a QA perspective beyond 'Project must have a basic devstack-gate job set up'. As a result, I hear different interpretations on what a basic devstack gate job is. This topic was discussed in one of the QA meetings a few weeks back [1]. Based on the discussion there, having a basic job - such as one that will let us know 'if a keystone change broke marconi' was good enough. My efforts in getting Marconi meet graduation requirements w.r.t Tempest was based on the above discussion. However, my conversations with the TC during Marconi's graduation review lead me to believe that these requirements aren't yet formalized. We were told that we needed to have more test coverage in tempest, having them elsewhere (i.e. functional tests in the Marconi project itself) was not good enough. So having only looked at the Marconi ML thread and not the actual TC meeting minutes I might be missing the whole picture. But, from what I saw when I looked at both a marconi commit and a tempest commit is that there is no gating marconi devstack-gate job on marconi commits. It's only non-voting in the check pipeline. Additionally, there isn't a non-voting job on tempest or
Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest
On Thu, Mar 20, 2014 at 1:19 PM, Rochelle.RochelleGrober rochelle.gro...@huawei.com wrote: -Original Message- From: Malini Kamalambal [mailto:malini.kamalam...@rackspace.com] Sent: Thursday, March 20, 2014 12:13 PM 'project specific functional testing' in the Marconi context is treating Marconi as a complete system, making Marconi API calls verifying the response - just like an end user would, but without keystone. If one of these tests fail, it is because there is a bug in the Marconi code , and not because its interaction with Keystone caused it to fail. That being said there are certain cases where having a project specific functional test makes sense. For example swift has a functional test job that starts swift in devstack. But, those things are normally handled on a per case basis. In general if the project is meant to be part of the larger OpenStack ecosystem then Tempest is the place to put functional testing. That way you know it works with all of the other components. The thing is in openstack what seems like a project isolated functional test almost always involves another project in real use cases. (for example keystone auth with api requests) One of the concerns we heard in the review was 'having the functional tests elsewhere (I.e within the project itself) does not count and they have to be in Tempest'. This has made us as a team wonder if we should migrate all our functional tests to Tempest. But from Matt's response, I think it is reasonable to continue in our current path have the functional tests in Marconi coexist along with the tests in Tempest. While there always exceptions to this rule, in general I think functional tests belong in tempest for all of the reasons discussed above. I think that what is being asked, really is that the functional tests could be a single set of tests that would become a part of the tempest repository and that these tests would have an ENV variable as part of the configuration that would allow either no Keystone or Keystone or some such, if that is the only configuration issue that separates running the tests isolated vs. integrated. The functional tests need to be as much as possible a single set of tests to reduce duplication and remove the likelihood of two sets getting out of sync with each other/development. If they only run in the integrated environment, that's ok, but if you want to run them isolated to make debugging easier, then it should be a configuration option and a separate test job. So, if my assumptions are correct, QA only requires functional tests for integrated runs, but if the project QAs/Devs want to run isolated for dev and devtest purposes, more power to them. Just keep it a single set of functional tests and put them in the Tempest repository so that if a failure happens, anyone can find the test and do the debug work without digging into a separate project repository. Hopefully, the tests as designed could easily take a new configuration directive and a short bit of work with OS QA will get the integrated FTs working as well as the isolated ones. --Rocky On 3/20/14 1:59 PM, Matthew Treinish mtrein...@kortar.org wrote: On Thu, Mar 20, 2014 at 11:35:15AM +, Malini Kamalambal wrote: Hello all, I have been working on adding tests in Tempest for Marconi, for the last few months. While there are many amazing people to work with, the process has been more difficult than I expected. Couple of pain-points and suggestions to make the process easier for myself future contributors. 1. The QA requirements for a project to graduate needs details beyond the Project must have a *basic* devstack-gate job set up 2. The scope of Tempest needs clarification - what tests should be in Tempest vs. in the individual projects? Or should they be in both tempest and the project? See details below. 1. There is little documentation on graduation requirement from a QA perspective beyond 'Project must have a basic devstack-gate job set up'. As a result, I hear different interpretations on what a basic devstack gate job is. This topic was discussed in one of the QA meetings a few weeks back [1]. Based on the discussion there, having a basic job - such as one that will let us know 'if a keystone change broke marconi' was good enough. My efforts in getting Marconi meet graduation requirements w.r.t Tempest was based on the above discussion. However, my conversations with the TC during Marconi's graduation review lead me to believe that these requirements aren't yet formalized. We were told that we needed to have more test coverage in tempest, having them elsewhere (i.e. functional tests in the Marconi project itself) was not good enough. So having only looked at the Marconi ML thread and not the actual TC
