Re: [openstack-dev] [QA][All] Prelude to functional testing summit discussions
Hi everyone, Before we start the larger discussion at summit next week about the future of testing in OpenStack - specifically about spinning up functional testing and how it relates to tempest - I would like to share some of my thoughts on how we can get things started and how I think they'll eventually come together. Currently in tempest we have a large number of tests (mostly api-focused) which are probably a better fit for a project's functional test suite. The best example I can think of is the nova flavors tests. Validation of flavor manipulation doesn't need to run in the integrated test suite on every commit to every project because it only requires Nova. A simple win for initiating in-tree functional testing would be to move these kinds of tests into the projects and run the tests from the project repos instead of from tempest. This would have the advantage of making tempest slimmer for every project and begin the process of getting projects to take responsibility for their functional testing rather than relying on tempest. As tests are moved tempest can start to become the integration test suite it was meant to be. It would retain only tests that involve multiple projects and stop being the OpenStack black box testing suite. I think that this is the right direction for tempest moving forward, especially as we move to having project-specific functional testing. Doing this migration is dependent on some refactors in tempest and moving the required bits to tempest-lib so they can be easily consumed by the other projects. This will be discussed at summit, is being planned for implementation this cycle, and is similar to what is currently in progress for the cli tests. The only reason this testing existed in tempest in the first place was as mechanism to block and then add friction against breaking api changes. Tempest's api testing has been been pretty successful at achieving these goals. We'll want to ensure that migrated tests retain these characteristics. If we are using clients from tempest-lib we should get this automatically since to break the api you'd have to change the api client. Another option proposed was to introduce a hacking rule that would block changes to api tests at the same time other code was being changed. There is also a concern for external consumers of tempest if we move the tests out of the tempest tree (I'm thinking refstack). I think the solution is to maintain a load_tests discovery method inside of tempest or elsewhere that will run the appropriate tests from the other repos for something like refstack. Assuming that things are built in a compatible way using the same framework then running the tests from separate repos should be a simple matter of pointing the test runner in the right direction. I also want to comment on the role of functional testing. What I've proposed here is only one piece of what project specific functional testing should be and just what I feel is a good/easy start. I don't feel that this should be the only testing done in the projects. I'm suggesting this as a first step because the tests already exist and it should be a relatively simple task. I also feel that using tempest-lib like this shouldn't be a hard requirement. Ideally the client definitions shouldn't have to live externally, or if they did they would be the official clients, but I am suggesting this as a first step to start a migration out of tempest. I don't want anyone to feel that they need block their functional testing efforts until tempest-lib becomes more useable. The larger value from functional testing is actually in enabling testing more tightly coupled to the projects (e.g. whitebox testing). I feel that any work necessary to enable functional testing should be prioritized. Thanks Matt for getting the ball rolling on this conversation in advance of summit. Probably stating the obvious here, but I feel we should make a concious effort to keep the approaches to in-tree functional testing as consistent as possible across projects. Towards that end, it would be good for folks with an interest in this area to attend each other's sessions where possible: Cross-project: Tue, 12:05 [1] Heat: Wed, 13:50 [2] Nova: Wed, 16:30 [3] Ceilometer:Wed, 17:20 [4] QA:Wed, 17:20 [5] Unfortunately there's a clash there between the QA tempest-lib session and the ceilo session. I'm not sure how reasonable it would be to make a last-minute schedule change to avoid that clash. Cheers, Eoghan [1] http://kilodesignsummit.sched.org/event/575938e4837e8293615845582d7e3e7f [2] http://kilodesignsummit.sched.org/event/eb261fb08b18ec1eaa2c67492e7fc385 [3] http://kilodesignsummit.sched.org/event/271a9075c1ced6c1269100ff4b8efc37 [4] http://kilodesignsummit.sched.org/event/daf63526a1883e84cec107c70cc6cad3 [5]
Re: [openstack-dev] [QA][All] Prelude to functional testing summit discussions
On 10/29/2014 12:30 PM, Matthew Treinish wrote: Hi everyone, Before we start the larger discussion at summit next week about the future of testing in OpenStack - specifically about spinning up functional testing and how it relates to tempest - I would like to share some of my thoughts on how we can get things started and how I think they'll eventually come together. Currently in tempest we have a large number of tests (mostly api-focused) which are probably a better fit for a project's functional test suite. The best example I can think of is the nova flavors tests. Validation of flavor manipulation doesn't need to run in the integrated test suite on every commit to every project because it only requires Nova. A simple win for initiating in-tree functional testing would be to move these kinds of tests into the projects and run the tests from the project repos instead of from tempest. I think a lot of the negative API testing is also a great thing to be done back at the project level. All of that testing should be able to work without a full OpenStack, as it should be caught and managed by the API service and never get any further than that. This would have the advantage of making tempest slimmer for every project and begin the process of getting projects to take responsibility for their functional testing rather than relying on tempest. As tests are moved tempest can start to become the integration test suite it was meant to be. It would retain only tests that involve multiple projects and stop being the OpenStack black box testing suite. I think that this is the right direction for tempest moving forward, especially as we move to having project-specific functional testing. Doing this migration is dependent on some refactors in tempest and moving the required bits to tempest-lib so they can be easily consumed by the other projects. This will be discussed at summit, is being planned for implementation this cycle, and is similar to what is currently in progress for the cli tests. The only reason this testing existed in tempest in the first place was as mechanism to block and then add friction against breaking api changes. Tempest's api testing has been been pretty successful at achieving these goals. We'll want to ensure that migrated tests retain these characteristics. If we are using clients from tempest-lib we should get this automatically since to break the api you'd have to change the api client. Another option proposed was to introduce a hacking rule that would block changes to api tests at the same time other code was being changed. There is also a concern for external consumers of tempest if we move the tests out of the tempest tree (I'm thinking refstack). I think the solution is to maintain a load_tests discovery method inside of tempest or elsewhere that will run the appropriate tests from the other repos for something like refstack. Assuming that things are built in a compatible way using the same framework then running the tests from separate repos should be a simple matter of pointing the test runner in the right direction. I think we can see where this takes us. I'm still skeptical of cross project loading of tests because it's often quite fragile. However, if you look at what refstack did they had a giant evaluation of all of tempest and pruned a bunch of stuff out. I would imagine maybe there is a conversation there about tests that refstack feels are important to stay in Tempest for their validation reasons. I think having a few paths that are tested both in Tempest and in project functional tests is not a bad thing. But I think that's an end of cycle at best discussion. Also, there probably need to be a few discussions anyway of refstack/tempest/defcore. The fact that Keystone was dropped from defcore because there were no non admin Keystone tests explicitly in Tempest (even though we make over 5000 keystone non admin API calls over a tempest run) was very odd. That is something that could have been fixed in a day. I also want to comment on the role of functional testing. What I've proposed here is only one piece of what project specific functional testing should be and just what I feel is a good/easy start. I don't feel that this should be the only testing done in the projects. I'm suggesting this as a first step because the tests already exist and it should be a relatively simple task. I also feel that using tempest-lib like this shouldn't be a hard requirement. Ideally the client definitions shouldn't have to live externally, or if they did they would be the official clients, but I am suggesting this as a first step to start a migration out of tempest. I don't want anyone to feel that they need block their functional testing efforts until tempest-lib becomes more useable. The larger value from functional testing is actually in enabling testing more tightly coupled to the projects (e.g.
Re: [openstack-dev] [QA][All] Prelude to functional testing summit discussions
On 10/30/2014 07:49 AM, Sean Dague wrote: On 10/29/2014 12:30 PM, Matthew Treinish wrote: Hi everyone, Before we start the larger discussion at summit next week about the future of testing in OpenStack - specifically about spinning up functional testing and how it relates to tempest - I would like to share some of my thoughts on how we can get things started and how I think they'll eventually come together. Currently in tempest we have a large number of tests (mostly api-focused) which are probably a better fit for a project's functional test suite. The best example I can think of is the nova flavors tests. Validation of flavor manipulation doesn't need to run in the integrated test suite on every commit to every project because it only requires Nova. A simple win for initiating in-tree functional testing would be to move these kinds of tests into the projects and run the tests from the project repos instead of from tempest. I think a lot of the negative API testing is also a great thing to be done back at the project level. All of that testing should be able to work without a full OpenStack, as it should be caught and managed by the API service and never get any further than that. This would have the advantage of making tempest slimmer for every project and begin the process of getting projects to take responsibility for their functional testing rather than relying on tempest. As tests are moved tempest can start to become the integration test suite it was meant to be. It would retain only tests that involve multiple projects and stop being the OpenStack black box testing suite. I think that this is the right direction for tempest moving forward, especially as we move to having project-specific functional testing. Doing this migration is dependent on some refactors in tempest and moving the required bits to tempest-lib so they can be easily consumed by the other projects. This will be discussed at summit, is being planned for implementation this cycle, and is similar to what is currently in progress for the cli tests. The only reason this testing existed in tempest in the first place was as mechanism to block and then add friction against breaking api changes. Tempest's api testing has been been pretty successful at achieving these goals. We'll want to ensure that migrated tests retain these characteristics. If we are using clients from tempest-lib we should get this automatically since to break the api you'd have to change the api client. Another option proposed was to introduce a hacking rule that would block changes to api tests at the same time other code was being changed. There is also a concern for external consumers of tempest if we move the tests out of the tempest tree (I'm thinking refstack). I think the solution is to maintain a load_tests discovery method inside of tempest or elsewhere that will run the appropriate tests from the other repos for something like refstack. Assuming that things are built in a compatible way using the same framework then running the tests from separate repos should be a simple matter of pointing the test runner in the right direction. I think we can see where this takes us. I'm still skeptical of cross project loading of tests because it's often quite fragile. However, if you look at what refstack did they had a giant evaluation of all of tempest and pruned a bunch of stuff out. I would imagine maybe there is a conversation there about tests that refstack feels are important to stay in Tempest for their validation reasons. I think having a few paths that are tested both in Tempest and in project functional tests is not a bad thing. Refstack is not the only thing that cares about validation of real clouds. As we move forward with this, it would be good to separate the issues of in which repo does a functional test live and can a functional test be run against a real cloud. IMO, over use of mocking (broadly defined) in functional tests should be avoided unless it is configurable to also work in an unmocked fashion. Whether the way to combine all of the functional tests is by cross project loading of tests or by some other means is more of an implementation detail. But I think that's an end of cycle at best discussion. Also, there probably need to be a few discussions anyway of refstack/tempest/defcore. The fact that Keystone was dropped from defcore because there were no non admin Keystone tests explicitly in Tempest (even though we make over 5000 keystone non admin API calls over a tempest run) was very odd. That is something that could have been fixed in a day. I also want to comment on the role of functional testing. What I've proposed here is only one piece of what project specific functional testing should be and just what I feel is a good/easy start. I don't feel that this should be the only testing done in the projects. I'm suggesting this as a first step because the tests already exist and it should be a relatively
Re: [openstack-dev] [QA][All] Prelude to functional testing summit discussions
On 10/30/2014 09:33 AM, David Kranz wrote: On 10/30/2014 07:49 AM, Sean Dague wrote: On 10/29/2014 12:30 PM, Matthew Treinish wrote: Hi everyone, Before we start the larger discussion at summit next week about the future of testing in OpenStack - specifically about spinning up functional testing and how it relates to tempest - I would like to share some of my thoughts on how we can get things started and how I think they'll eventually come together. Currently in tempest we have a large number of tests (mostly api-focused) which are probably a better fit for a project's functional test suite. The best example I can think of is the nova flavors tests. Validation of flavor manipulation doesn't need to run in the integrated test suite on every commit to every project because it only requires Nova. A simple win for initiating in-tree functional testing would be to move these kinds of tests into the projects and run the tests from the project repos instead of from tempest. I think a lot of the negative API testing is also a great thing to be done back at the project level. All of that testing should be able to work without a full OpenStack, as it should be caught and managed by the API service and never get any further than that. This would have the advantage of making tempest slimmer for every project and begin the process of getting projects to take responsibility for their functional testing rather than relying on tempest. As tests are moved tempest can start to become the integration test suite it was meant to be. It would retain only tests that involve multiple projects and stop being the OpenStack black box testing suite. I think that this is the right direction for tempest moving forward, especially as we move to having project-specific functional testing. Doing this migration is dependent on some refactors in tempest and moving the required bits to tempest-lib so they can be easily consumed by the other projects. This will be discussed at summit, is being planned for implementation this cycle, and is similar to what is currently in progress for the cli tests. The only reason this testing existed in tempest in the first place was as mechanism to block and then add friction against breaking api changes. Tempest's api testing has been been pretty successful at achieving these goals. We'll want to ensure that migrated tests retain these characteristics. If we are using clients from tempest-lib we should get this automatically since to break the api you'd have to change the api client. Another option proposed was to introduce a hacking rule that would block changes to api tests at the same time other code was being changed. There is also a concern for external consumers of tempest if we move the tests out of the tempest tree (I'm thinking refstack). I think the solution is to maintain a load_tests discovery method inside of tempest or elsewhere that will run the appropriate tests from the other repos for something like refstack. Assuming that things are built in a compatible way using the same framework then running the tests from separate repos should be a simple matter of pointing the test runner in the right direction. I think we can see where this takes us. I'm still skeptical of cross project loading of tests because it's often quite fragile. However, if you look at what refstack did they had a giant evaluation of all of tempest and pruned a bunch of stuff out. I would imagine maybe there is a conversation there about tests that refstack feels are important to stay in Tempest for their validation reasons. I think having a few paths that are tested both in Tempest and in project functional tests is not a bad thing. Refstack is not the only thing that cares about validation of real clouds. As we move forward with this, it would be good to separate the issues of in which repo does a functional test live and can a functional test be run against a real cloud. IMO, over use of mocking (broadly defined) in functional tests should be avoided unless it is configurable to also work in an unmocked fashion. Whether the way to combine all of the functional tests is by cross project loading of tests or by some other means is more of an implementation detail. Part of the perspective I'm bringing in is actually knowing what to do when your tests fail. Using Tempest against real clouds is great, people should keep doing that. But if you are rolling out a real cloud yourself, in the future you should be running the functional tests in staging to ensure you are functioning. Those will also provide you, hopefully, with a better path to understand what's wrong. This will mean that as an arbitrary 3rd party accessing a public cloud, you don't have a test suite that pushes every button of the cloud. But I think that's an acceptable trade off. Because pushing on odd edge cases might produce a fail, but if there is no
Re: [openstack-dev] [QA][All] Prelude to functional testing summit discussions
On 10/30/2014 09:52 AM, Sean Dague wrote: On 10/30/2014 09:33 AM, David Kranz wrote: On 10/30/2014 07:49 AM, Sean Dague wrote: On 10/29/2014 12:30 PM, Matthew Treinish wrote: Hi everyone, Before we start the larger discussion at summit next week about the future of testing in OpenStack - specifically about spinning up functional testing and how it relates to tempest - I would like to share some of my thoughts on how we can get things started and how I think they'll eventually come together. Currently in tempest we have a large number of tests (mostly api-focused) which are probably a better fit for a project's functional test suite. The best example I can think of is the nova flavors tests. Validation of flavor manipulation doesn't need to run in the integrated test suite on every commit to every project because it only requires Nova. A simple win for initiating in-tree functional testing would be to move these kinds of tests into the projects and run the tests from the project repos instead of from tempest. I think a lot of the negative API testing is also a great thing to be done back at the project level. All of that testing should be able to work without a full OpenStack, as it should be caught and managed by the API service and never get any further than that. This would have the advantage of making tempest slimmer for every project and begin the process of getting projects to take responsibility for their functional testing rather than relying on tempest. As tests are moved tempest can start to become the integration test suite it was meant to be. It would retain only tests that involve multiple projects and stop being the OpenStack black box testing suite. I think that this is the right direction for tempest moving forward, especially as we move to having project-specific functional testing. Doing this migration is dependent on some refactors in tempest and moving the required bits to tempest-lib so they can be easily consumed by the other projects. This will be discussed at summit, is being planned for implementation this cycle, and is similar to what is currently in progress for the cli tests. The only reason this testing existed in tempest in the first place was as mechanism to block and then add friction against breaking api changes. Tempest's api testing has been been pretty successful at achieving these goals. We'll want to ensure that migrated tests retain these characteristics. If we are using clients from tempest-lib we should get this automatically since to break the api you'd have to change the api client. Another option proposed was to introduce a hacking rule that would block changes to api tests at the same time other code was being changed. There is also a concern for external consumers of tempest if we move the tests out of the tempest tree (I'm thinking refstack). I think the solution is to maintain a load_tests discovery method inside of tempest or elsewhere that will run the appropriate tests from the other repos for something like refstack. Assuming that things are built in a compatible way using the same framework then running the tests from separate repos should be a simple matter of pointing the test runner in the right direction. I think we can see where this takes us. I'm still skeptical of cross project loading of tests because it's often quite fragile. However, if you look at what refstack did they had a giant evaluation of all of tempest and pruned a bunch of stuff out. I would imagine maybe there is a conversation there about tests that refstack feels are important to stay in Tempest for their validation reasons. I think having a few paths that are tested both in Tempest and in project functional tests is not a bad thing. Refstack is not the only thing that cares about validation of real clouds. As we move forward with this, it would be good to separate the issues of in which repo does a functional test live and can a functional test be run against a real cloud. IMO, over use of mocking (broadly defined) in functional tests should be avoided unless it is configurable to also work in an unmocked fashion. Whether the way to combine all of the functional tests is by cross project loading of tests or by some other means is more of an implementation detail. Part of the perspective I'm bringing in is actually knowing what to do when your tests fail. Using Tempest against real clouds is great, people should keep doing that. But if you are rolling out a real cloud yourself, in the future you should be running the functional tests in staging to ensure you are functioning. Those will also provide you, hopefully, with a better path to understand what's wrong. Sean, sorry if I was unclear. By real clouds, I just meant the tests should be able to use OpenStack apis with no mocking. -David This will mean that as an arbitrary 3rd party accessing a public cloud, you don't have a test suite that pushes every button of the cloud. But I
Re: [openstack-dev] [QA][All] Prelude to functional testing summit discussions
On 10/30/2014 10:02 AM, David Kranz wrote: On 10/30/2014 09:52 AM, Sean Dague wrote: On 10/30/2014 09:33 AM, David Kranz wrote: On 10/30/2014 07:49 AM, Sean Dague wrote: On 10/29/2014 12:30 PM, Matthew Treinish wrote: Hi everyone, Before we start the larger discussion at summit next week about the future of testing in OpenStack - specifically about spinning up functional testing and how it relates to tempest - I would like to share some of my thoughts on how we can get things started and how I think they'll eventually come together. Currently in tempest we have a large number of tests (mostly api-focused) which are probably a better fit for a project's functional test suite. The best example I can think of is the nova flavors tests. Validation of flavor manipulation doesn't need to run in the integrated test suite on every commit to every project because it only requires Nova. A simple win for initiating in-tree functional testing would be to move these kinds of tests into the projects and run the tests from the project repos instead of from tempest. I think a lot of the negative API testing is also a great thing to be done back at the project level. All of that testing should be able to work without a full OpenStack, as it should be caught and managed by the API service and never get any further than that. This would have the advantage of making tempest slimmer for every project and begin the process of getting projects to take responsibility for their functional testing rather than relying on tempest. As tests are moved tempest can start to become the integration test suite it was meant to be. It would retain only tests that involve multiple projects and stop being the OpenStack black box testing suite. I think that this is the right direction for tempest moving forward, especially as we move to having project-specific functional testing. Doing this migration is dependent on some refactors in tempest and moving the required bits to tempest-lib so they can be easily consumed by the other projects. This will be discussed at summit, is being planned for implementation this cycle, and is similar to what is currently in progress for the cli tests. The only reason this testing existed in tempest in the first place was as mechanism to block and then add friction against breaking api changes. Tempest's api testing has been been pretty successful at achieving these goals. We'll want to ensure that migrated tests retain these characteristics. If we are using clients from tempest-lib we should get this automatically since to break the api you'd have to change the api client. Another option proposed was to introduce a hacking rule that would block changes to api tests at the same time other code was being changed. There is also a concern for external consumers of tempest if we move the tests out of the tempest tree (I'm thinking refstack). I think the solution is to maintain a load_tests discovery method inside of tempest or elsewhere that will run the appropriate tests from the other repos for something like refstack. Assuming that things are built in a compatible way using the same framework then running the tests from separate repos should be a simple matter of pointing the test runner in the right direction. I think we can see where this takes us. I'm still skeptical of cross project loading of tests because it's often quite fragile. However, if you look at what refstack did they had a giant evaluation of all of tempest and pruned a bunch of stuff out. I would imagine maybe there is a conversation there about tests that refstack feels are important to stay in Tempest for their validation reasons. I think having a few paths that are tested both in Tempest and in project functional tests is not a bad thing. Refstack is not the only thing that cares about validation of real clouds. As we move forward with this, it would be good to separate the issues of in which repo does a functional test live and can a functional test be run against a real cloud. IMO, over use of mocking (broadly defined) in functional tests should be avoided unless it is configurable to also work in an unmocked fashion. Whether the way to combine all of the functional tests is by cross project loading of tests or by some other means is more of an implementation detail. Part of the perspective I'm bringing in is actually knowing what to do when your tests fail. Using Tempest against real clouds is great, people should keep doing that. But if you are rolling out a real cloud yourself, in the future you should be running the functional tests in staging to ensure you are functioning. Those will also provide you, hopefully, with a better path to understand what's wrong. Sean, sorry if I was unclear. By real clouds, I just meant the tests should be able to use OpenStack apis with no mocking. Ok, so part of this remains to be
Re: [openstack-dev] [QA][All] Prelude to functional testing summit discussions
On Thu, Oct 30, 2014 at 6:30 AM, Eoghan Glynn egl...@redhat.com wrote: Hi everyone, Before we start the larger discussion at summit next week about the future of testing in OpenStack - specifically about spinning up functional testing and how it relates to tempest - I would like to share some of my thoughts on how we can get things started and how I think they'll eventually come together. Currently in tempest we have a large number of tests (mostly api-focused) which are probably a better fit for a project's functional test suite. The best example I can think of is the nova flavors tests. Validation of flavor manipulation doesn't need to run in the integrated test suite on every commit to every project because it only requires Nova. A simple win for initiating in-tree functional testing would be to move these kinds of tests into the projects and run the tests from the project repos instead of from tempest. This would have the advantage of making tempest slimmer for every project and begin the process of getting projects to take responsibility for their functional testing rather than relying on tempest. As tests are moved tempest can start to become the integration test suite it was meant to be. It would retain only tests that involve multiple projects and stop being the OpenStack black box testing suite. I think that this is the right direction for tempest moving forward, especially as we move to having project-specific functional testing. Doing this migration is dependent on some refactors in tempest and moving the required bits to tempest-lib so they can be easily consumed by the other projects. This will be discussed at summit, is being planned for implementation this cycle, and is similar to what is currently in progress for the cli tests. The only reason this testing existed in tempest in the first place was as mechanism to block and then add friction against breaking api changes. Tempest's api testing has been been pretty successful at achieving these goals. We'll want to ensure that migrated tests retain these characteristics. If we are using clients from tempest-lib we should get this automatically since to break the api you'd have to change the api client. Another option proposed was to introduce a hacking rule that would block changes to api tests at the same time other code was being changed. There is also a concern for external consumers of tempest if we move the tests out of the tempest tree (I'm thinking refstack). I think the solution is to maintain a load_tests discovery method inside of tempest or elsewhere that will run the appropriate tests from the other repos for something like refstack. Assuming that things are built in a compatible way using the same framework then running the tests from separate repos should be a simple matter of pointing the test runner in the right direction. I also want to comment on the role of functional testing. What I've proposed here is only one piece of what project specific functional testing should be and just what I feel is a good/easy start. I don't feel that this should be the only testing done in the projects. I'm suggesting this as a first step because the tests already exist and it should be a relatively simple task. I also feel that using tempest-lib like this shouldn't be a hard requirement. Ideally the client definitions shouldn't have to live externally, or if they did they would be the official clients, but I am suggesting this as a first step to start a migration out of tempest. I don't want anyone to feel that they need block their functional testing efforts until tempest-lib becomes more useable. The larger value from functional testing is actually in enabling testing more tightly coupled to the projects (e.g. whitebox testing). I feel that any work necessary to enable functional testing should be prioritized. Thanks Matt for getting the ball rolling on this conversation in advance of summit. Probably stating the obvious here, but I feel we should make a concious effort to keep the approaches to in-tree functional testing as consistent as possible across projects. Towards that end, it would be good for folks with an interest in this area to attend each other's sessions where possible: +1 Cross-project: Tue, 12:05 [1] Heat: Wed, 13:50 [2] Nova: Wed, 16:30 [3] Ceilometer:Wed, 17:20 [4] QA:Wed, 17:20 [5] I think Keystone was planning on dedicating some time to this on Friday, so our dev/hackathon day. I'm interested in the process that will be in place for tracking status of the functional test migration if there will be one. Unfortunately there's a clash there between the QA tempest-lib session and the ceilo session. I'm not sure how reasonable it would be to make a last-minute schedule
Re: [openstack-dev] [QA][All] Prelude to functional testing summit discussions
Matthew wrote: This would have the advantage of making tempest slimmer for every project and begin the process of getting projects to take responsibility for their functional testing rather than relying on tempest. [much snipping] Sean wrote: Ok, so part of this remains to be seen about what the biggest bang for the buck is. The class of bugs I feel like we need to nail in Nova right now are going to require tests that bring up pieces of the wsgi stack, but are probably not runable on a real deploy. Again, this is about debugability. So this notion of the biggest bang for our buck is an aspect of the drive for in-tree functional tests, that's not entirely clear to me as yet. i.e. whether individual projects should be prioritizing within this effort: (a) the creation of net-new coverage for scenarios (especially known or suspected bugs) that were not previously tested, in a non-unit sense (b) the relocation of existing integration test coverage from Tempest to the project trees, in order to make the management of Tempest more tractable It feels like there may be a tension between (a) and (b) in terms of the pay-off for this effort. I'd interested in hearing other opinions on this, on what aspect projects are expecting (and expected) to concentrate on initially. Cheers, Eoghan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [QA][All] Prelude to functional testing summit discussions
On 10/30/2014 10:47 AM, Eoghan Glynn wrote: Matthew wrote: This would have the advantage of making tempest slimmer for every project and begin the process of getting projects to take responsibility for their functional testing rather than relying on tempest. [much snipping] Sean wrote: Ok, so part of this remains to be seen about what the biggest bang for the buck is. The class of bugs I feel like we need to nail in Nova right now are going to require tests that bring up pieces of the wsgi stack, but are probably not runable on a real deploy. Again, this is about debugability. So this notion of the biggest bang for our buck is an aspect of the drive for in-tree functional tests, that's not entirely clear to me as yet. i.e. whether individual projects should be prioritizing within this effort: (a) the creation of net-new coverage for scenarios (especially known or suspected bugs) that were not previously tested, in a non-unit sense (b) the relocation of existing integration test coverage from Tempest to the project trees, in order to make the management of Tempest more tractable It feels like there may be a tension between (a) and (b) in terms of the pay-off for this effort. I'd interested in hearing other opinions on this, on what aspect projects are expecting (and expected) to concentrate on initially. For what it's worth I have a bunch of early targets listed for Nova for our summit session - https://etherpad.openstack.org/p/kilo-nova-functional-testing My focus in kilo is going to be first about A), as that provides value out of the gate (pun intended). Then peel off some stuff from B as makes sense. -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [QA][All] Prelude to functional testing summit discussions
On 10/30/2014 11:12 AM, Sean Dague wrote: On 10/30/2014 10:47 AM, Eoghan Glynn wrote: Matthew wrote: This would have the advantage of making tempest slimmer for every project and begin the process of getting projects to take responsibility for their functional testing rather than relying on tempest. [much snipping] Sean wrote: Ok, so part of this remains to be seen about what the biggest bang for the buck is. The class of bugs I feel like we need to nail in Nova right now are going to require tests that bring up pieces of the wsgi stack, but are probably not runable on a real deploy. Again, this is about debugability. So this notion of the biggest bang for our buck is an aspect of the drive for in-tree functional tests, that's not entirely clear to me as yet. i.e. whether individual projects should be prioritizing within this effort: (a) the creation of net-new coverage for scenarios (especially known or suspected bugs) that were not previously tested, in a non-unit sense (b) the relocation of existing integration test coverage from Tempest to the project trees, in order to make the management of Tempest more tractable It feels like there may be a tension between (a) and (b) in terms of the pay-off for this effort. I'd interested in hearing other opinions on this, on what aspect projects are expecting (and expected) to concentrate on initially. For what it's worth I have a bunch of early targets listed for Nova for our summit session - https://etherpad.openstack.org/p/kilo-nova-functional-testing My focus in kilo is going to be first about A), as that provides value out of the gate (pun intended). Then peel off some stuff from B as makes sense. -Sean That seems sensible from the nova point of view and overall health, and not all projects have to pursue the same priorities at the same time. But a big part of the benefit of (b) is the impact it has on all the other projects in that other projects will stop getting as many gate failures, and that benefit could be achieved right now by simply changing the set of tempest tests that run against each project. -David ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [QA][All] Prelude to functional testing summit discussions
Hi everyone, Before we start the larger discussion at summit next week about the future of testing in OpenStack - specifically about spinning up functional testing and how it relates to tempest - I would like to share some of my thoughts on how we can get things started and how I think they'll eventually come together. Currently in tempest we have a large number of tests (mostly api-focused) which are probably a better fit for a project's functional test suite. The best example I can think of is the nova flavors tests. Validation of flavor manipulation doesn't need to run in the integrated test suite on every commit to every project because it only requires Nova. A simple win for initiating in-tree functional testing would be to move these kinds of tests into the projects and run the tests from the project repos instead of from tempest. This would have the advantage of making tempest slimmer for every project and begin the process of getting projects to take responsibility for their functional testing rather than relying on tempest. As tests are moved tempest can start to become the integration test suite it was meant to be. It would retain only tests that involve multiple projects and stop being the OpenStack black box testing suite. I think that this is the right direction for tempest moving forward, especially as we move to having project-specific functional testing. Doing this migration is dependent on some refactors in tempest and moving the required bits to tempest-lib so they can be easily consumed by the other projects. This will be discussed at summit, is being planned for implementation this cycle, and is similar to what is currently in progress for the cli tests. The only reason this testing existed in tempest in the first place was as mechanism to block and then add friction against breaking api changes. Tempest's api testing has been been pretty successful at achieving these goals. We'll want to ensure that migrated tests retain these characteristics. If we are using clients from tempest-lib we should get this automatically since to break the api you'd have to change the api client. Another option proposed was to introduce a hacking rule that would block changes to api tests at the same time other code was being changed. There is also a concern for external consumers of tempest if we move the tests out of the tempest tree (I'm thinking refstack). I think the solution is to maintain a load_tests discovery method inside of tempest or elsewhere that will run the appropriate tests from the other repos for something like refstack. Assuming that things are built in a compatible way using the same framework then running the tests from separate repos should be a simple matter of pointing the test runner in the right direction. I also want to comment on the role of functional testing. What I've proposed here is only one piece of what project specific functional testing should be and just what I feel is a good/easy start. I don't feel that this should be the only testing done in the projects. I'm suggesting this as a first step because the tests already exist and it should be a relatively simple task. I also feel that using tempest-lib like this shouldn't be a hard requirement. Ideally the client definitions shouldn't have to live externally, or if they did they would be the official clients, but I am suggesting this as a first step to start a migration out of tempest. I don't want anyone to feel that they need block their functional testing efforts until tempest-lib becomes more useable. The larger value from functional testing is actually in enabling testing more tightly coupled to the projects (e.g. whitebox testing). I feel that any work necessary to enable functional testing should be prioritized. -Matt Treinish pgpE5i_OwYONG.pgp Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [QA][All] Prelude to functional testing summit discussions
Hi Matt, Thanks for bringing this up, I am so interested in. 2014-10-30 1:30 GMT+09:00 Matthew Treinish mtrein...@kortar.org: Before we start the larger discussion at summit next week about the future of testing in OpenStack - specifically about spinning up functional testing and how it relates to tempest - I would like to share some of my thoughts on how we can get things started and how I think they'll eventually come together. Currently in tempest we have a large number of tests (mostly api-focused) which are probably a better fit for a project's functional test suite. The best example I can think of is the nova flavors tests. Validation of flavor manipulation doesn't need to run in the integrated test suite on every commit to every project because it only requires Nova. A simple win for initiating in-tree functional testing would be to move these kinds of tests into the projects and run the tests from the project repos instead of from tempest. I completely agree with the above comment. There is a lot of hard-coded negative tests in Tempest and most projects block these negative cases at their surfaces, these cases don't seem integrated tests. In addition, sometimes there are more corner cases in Tempest than each project. So it would be very nice to move this kind of tests from Tempest to each project and review them on the project. This would have the advantage of making tempest slimmer for every project and begin the process of getting projects to take responsibility for their functional testing rather than relying on tempest. As tests are moved tempest can start to become the integration test suite it was meant to be. It would retain only tests that involve multiple projects and stop being the OpenStack black box testing suite. I think that this is the right direction for tempest moving forward, especially as we move to having project-specific functional testing. +1 Doing this migration is dependent on some refactors in tempest and moving the required bits to tempest-lib so they can be easily consumed by the other projects. This will be discussed at summit, is being planned for implementation this cycle, and is similar to what is currently in progress for the cli tests. The only reason this testing existed in tempest in the first place was as mechanism to block and then add friction against breaking api changes. Tempest's api testing has been been pretty successful at achieving these goals. We'll want to ensure that migrated tests retain these characteristics. If we are using clients from tempest-lib we should get this automatically since to break the api you'd have to change the api client. Another option proposed was to introduce a hacking rule that would block changes to api tests at the same time other code was being changed. There is also a concern for external consumers of tempest if we move the tests out of the tempest tree (I'm thinking refstack). I think the solution is to maintain a load_tests discovery method inside of tempest or elsewhere that will run the appropriate tests from the other repos for something like refstack. Assuming that things are built in a compatible way using the same framework then running the tests from separate repos should be a simple matter of pointing the test runner in the right direction. I also want to comment on the role of functional testing. What I've proposed here is only one piece of what project specific functional testing should be and just what I feel is a good/easy start. I don't feel that this should be the only testing done in the projects. I'm suggesting this as a first step because the tests already exist and it should be a relatively simple task. I also feel that using tempest-lib like this shouldn't be a hard requirement. Ideally the client definitions shouldn't have to live externally, or if they did they would be the official clients, but I am suggesting this as a first step to start a migration out of tempest. +1 for moving functional tests which means non-cross projects tests to each project repo. I like to jump into this work, and I want to move negative tests to each project ASAP. Thanks Ken Ohmichi ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
