[Openstack-qa-team] IMPORTANT: Moratorium on adding negative test cases
QAers, There has been a pile-up of code reviews recently, and I wanted to explain a decision that a number of core QA team members reached last week and why some patchsets have not been reviewed. Two of the goals of Tempest are to have a functional integration test suite that stresses *different* things than the unit tests and runs in a reasonable amount of time (< 20 minutes or so). While it's been great to see the large influx of negative tests added recently to Tempest, there is a concern that the incremental value these tests might add does not counterbalance the lengthy times that some of the tests take to run. It was the advice of these core team members (including myself), that we put a moratorium on adding new negative tests to Tempest at this point and look instead at doing the following: * Determine which negative test cases that are currently in Tempest DO NOT exist in the unit test suite of the corresponding core project, add those unit tests appropriately, and then remove the negative tests from Tempest * Use a fuzz-testing grammar-based tool such as randgen [1] to do negative testing of the APIs [2] * Focus all energies on writing test cases that stress the integration test points between the services and do more than just API-level validation -- in other words, doing more SSH-ing into an instance for verification of networking setups, init files, etc * Change the way we are currently using Launchpad's bugs and blueprints system in the following ways: (1) No longer have bugs that stay around forever, with lists of new small tests to be added. Instead, bugs should describe a very specific task, in detail, and should be closed once the test is added to Tempest (2) Have someone dedicated to Bug Triaging and cleaning up the current list of open bugs -- I believe David Kranz and I will be tackling this. Please respond to this post if you have any questions about the important points above. We need to refocus the QA team a bit to add greater value and make Tempest the integration test suite that the other core projects can rely on. All the best, -jay [1] https://launchpad.net/randgen [2] https://lists.launchpad.net/openstack-qa-team/msg00155.html -- Mailing list: https://launchpad.net/~openstack-qa-team Post to : openstack-qa-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack-qa-team More help : https://help.launchpad.net/ListHelp
Re: [Openstack-qa-team] Issues with soft reboot
Daryl, I agree. I sent this message after seeing some more soft reboot tests being posted. What I meant was that we should limit soft reboot tests to the ones that are testing that functionality specifically, as opposed to tests that try to do something or other during reboot. -David On 6/19/2012 11:05 AM, Daryl Walleck wrote: Hi David, From a time perspective I see what you're saying. However, there's an important bit of functionality that is getting tested here: the fact that the soft reboot works regardless of hyper visor. I've always aimed to make Tempest hyper visor agnostic, and I would be hesitant to skip a valid test case. I think it's at least worth noting down as something we can revisit later, but I think there are other areas we can improve performance in first. Daryl Sent from my iPad On Jun 19, 2012, at 8:01 AM, "David Kranz" wrote: To help with the effort of making the Tempest suite run faster, we should avoid or skip the use of soft reboot in any tests, at least for now. The problem is that, according to Vish, soft reboot requires guest support. If the booted image doesn't have it, compute will wait (two minutes by default), and do a hard reboot. So right now almost all tests that do a soft reboot will take at least 150 seconds or so and will not actually be testing anything useful. There should be a soft reboot test that uses an image with guest support. -David References: https://bugs.launchpad.net/nova/+bug/1013747 https://bugs.launchpad.net/tempest/+bug/1014647 -- Mailing list: https://launchpad.net/~openstack-qa-team Post to : openstack-qa-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack-qa-team More help : https://help.launchpad.net/ListHelp -- Mailing list: https://launchpad.net/~openstack-qa-team Post to : openstack-qa-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack-qa-team More help : https://help.launchpad.net/ListHelp
Re: [Openstack-qa-team] Issues with soft reboot
Hi David, >From a time perspective I see what you're saying. However, there's an >important bit of functionality that is getting tested here: the fact that the >soft reboot works regardless of hyper visor. I've always aimed to make Tempest >hyper visor agnostic, and I would be hesitant to skip a valid test case. I >think it's at least worth noting down as something we can revisit later, but I >think there are other areas we can improve performance in first. Daryl Sent from my iPad On Jun 19, 2012, at 8:01 AM, "David Kranz" wrote: > To help with the effort of making the Tempest suite run faster, we should > avoid or skip the use of soft reboot in any tests, at least for now. The > problem is that, according to Vish, soft reboot requires guest support. If > the booted image doesn't have it, compute will wait (two minutes by default), > and do a hard reboot. So right now almost all tests that do a soft reboot > will take at least 150 seconds or so and will not actually be testing > anything useful. There should be a soft reboot test that uses an image with > guest support. > > -David > > References: > https://bugs.launchpad.net/nova/+bug/1013747 > https://bugs.launchpad.net/tempest/+bug/1014647 > > -- > Mailing list: https://launchpad.net/~openstack-qa-team > Post to : openstack-qa-team@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack-qa-team > More help : https://help.launchpad.net/ListHelp -- Mailing list: https://launchpad.net/~openstack-qa-team Post to : openstack-qa-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack-qa-team More help : https://help.launchpad.net/ListHelp
[Openstack-qa-team] Issues with soft reboot
To help with the effort of making the Tempest suite run faster, we should avoid or skip the use of soft reboot in any tests, at least for now. The problem is that, according to Vish, soft reboot requires guest support. If the booted image doesn't have it, compute will wait (two minutes by default), and do a hard reboot. So right now almost all tests that do a soft reboot will take at least 150 seconds or so and will not actually be testing anything useful. There should be a soft reboot test that uses an image with guest support. -David References: https://bugs.launchpad.net/nova/+bug/1013747 https://bugs.launchpad.net/tempest/+bug/1014647 -- Mailing list: https://launchpad.net/~openstack-qa-team Post to : openstack-qa-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack-qa-team More help : https://help.launchpad.net/ListHelp
