Re: [openstack-dev] [qa] trimming down Tempest smoke tag
On 1 May 2015 at 05:39, Sean Dague s...@dague.net wrote: On 04/30/2015 12:53 PM, Matt Riedemann wrote: snip Yeah why would you not include admin tests? Like listing services and hosts in nova? It is extremely easy to say why not include *, it's valid function and we're going to end up with an hour long smoke test. At some point, you have to cut. Nova... can you boot a server, cool. Can it get on the network? Glance, can I feed you an image and it looks like it works. Great. Keystone, you do something with users right, can I add one? There are so many features in OpenStack that we need to be aggressive on keeping this paired down. My thought is Smoke is not intended to be validation your cloud is working. It's validation that the cloud isn't a giant pile of fail. It might be a medium pile of fail, but some basic ops are working, so that's cool. +1 That said, I think having *a* admin call is worthwhile in smoke, because that (privileges working) is a good representative example of the broad ways in which things could be systemically broken. But it needed be explicit: If admin calls are implicit in setting up user tests, that would be more than sufficient IMO. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] trimming down Tempest smoke tag
On 04/30/2015 12:53 PM, Matt Riedemann wrote: snip Yeah why would you not include admin tests? Like listing services and hosts in nova? It is extremely easy to say why not include *, it's valid function and we're going to end up with an hour long smoke test. At some point, you have to cut. Nova... can you boot a server, cool. Can it get on the network? Glance, can I feed you an image and it looks like it works. Great. Keystone, you do something with users right, can I add one? There are so many features in OpenStack that we need to be aggressive on keeping this paired down. My thought is Smoke is not intended to be validation your cloud is working. It's validation that the cloud isn't a giant pile of fail. It might be a medium pile of fail, but some basic ops are working, so that's cool. -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] trimming down Tempest smoke tag
On 4/28/2015 4:19 PM, David Kranz wrote: On 04/28/2015 06:38 AM, Sean Dague wrote: The Tempest Smoke tag was originally introduced to provide a quick view of your OpenStack environment to ensure that a few basic things were working. It was intended to be fast. However, during Icehouse the smoke tag was repurposed as a way to let neutron not backslide (so it's massively overloaded with network tests). It current runs at about 15 minutes on neutron jobs. This is why grenade neutron takes *so* long, because we run tempest smoke twice. The smoke tag needs a diet. I believe our working definition should be something as follows: - Total run time should be fast (= 5 minutes) - No negative tests - No admin tests - No tests that test optional extensions - No tests that test advanced services (like lbaas, vpnaas) - No proxy service tests The criteria for a good set of tests is CRUD operations on basic services. For instance, with compute we should have built a few servers, ensure we can shut them down. For neutron we should have done some basic network / port plugging. That makes sense. On IRC, Sean and I agreed that this would include creation of users, projects, etc. So some of the keystone smoke tests will be left in even though admin. IMO, it is debatable whether admin is relevant as part of the criteria for smoke. We also previously had the 'smoke' tag include all of the scenario tests, which was fine when we had 6 scenario tests. However as those have grown I think that should be trimmed back to a few basic through scenarios. The results of this are - https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:smoke,n,z The impacts on our upstream gate will mean that grenade jobs will speed up dramatically (20 minutes faster on grenade neutron). There is one edge condition which exists, which is the check-tempest-dsvm-neutron-icehouse job. Neutron couldn't pass either a full or parallel tempest run in icehouse (it's far too racy). So that's current running the smoke-serial tag. This would end up reducing the number of tests run on that job. However, based on the number of rechecks I've had to run in this series, that job is currently at about a 30% fail rate - http://goo.gl/N2w7qc - which means some test reduction is probably in order anyway, as it's mostly just preventing other people from landing unrelated patches. This was something we were originally planning on doing during the QA Sprint but ran out of time. It looks like we'll plan to land this right after Tempest 4 is cut this week, so that people that really want the old behavior can stay on the Tempest 4 release, but master is moving forward. I think that once we trim down we can decide to point add specific tests later. I expect smoke to be a bit more fluid over time, so it's not a tag that anyone should count on a test going into that tag and staying forever. Agreed. The criteria and purpose should stay the same but individual tests may be added or removed from smoke. Thanks for doing this. -David -Sean __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Yeah why would you not include admin tests? Like listing services and hosts in nova? -- Thanks, Matt Riedemann __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] trimming down Tempest smoke tag
On 04/28/2015 06:38 AM, Sean Dague wrote: The Tempest Smoke tag was originally introduced to provide a quick view of your OpenStack environment to ensure that a few basic things were working. It was intended to be fast. However, during Icehouse the smoke tag was repurposed as a way to let neutron not backslide (so it's massively overloaded with network tests). It current runs at about 15 minutes on neutron jobs. This is why grenade neutron takes *so* long, because we run tempest smoke twice. The smoke tag needs a diet. I believe our working definition should be something as follows: - Total run time should be fast (= 5 minutes) - No negative tests - No admin tests - No tests that test optional extensions - No tests that test advanced services (like lbaas, vpnaas) - No proxy service tests The criteria for a good set of tests is CRUD operations on basic services. For instance, with compute we should have built a few servers, ensure we can shut them down. For neutron we should have done some basic network / port plugging. That makes sense. On IRC, Sean and I agreed that this would include creation of users, projects, etc. So some of the keystone smoke tests will be left in even though admin. IMO, it is debatable whether admin is relevant as part of the criteria for smoke. We also previously had the 'smoke' tag include all of the scenario tests, which was fine when we had 6 scenario tests. However as those have grown I think that should be trimmed back to a few basic through scenarios. The results of this are - https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:smoke,n,z The impacts on our upstream gate will mean that grenade jobs will speed up dramatically (20 minutes faster on grenade neutron). There is one edge condition which exists, which is the check-tempest-dsvm-neutron-icehouse job. Neutron couldn't pass either a full or parallel tempest run in icehouse (it's far too racy). So that's current running the smoke-serial tag. This would end up reducing the number of tests run on that job. However, based on the number of rechecks I've had to run in this series, that job is currently at about a 30% fail rate - http://goo.gl/N2w7qc - which means some test reduction is probably in order anyway, as it's mostly just preventing other people from landing unrelated patches. This was something we were originally planning on doing during the QA Sprint but ran out of time. It looks like we'll plan to land this right after Tempest 4 is cut this week, so that people that really want the old behavior can stay on the Tempest 4 release, but master is moving forward. I think that once we trim down we can decide to point add specific tests later. I expect smoke to be a bit more fluid over time, so it's not a tag that anyone should count on a test going into that tag and staying forever. Agreed. The criteria and purpose should stay the same but individual tests may be added or removed from smoke. Thanks for doing this. -David -Sean __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [qa] trimming down Tempest smoke tag
The Tempest Smoke tag was originally introduced to provide a quick view of your OpenStack environment to ensure that a few basic things were working. It was intended to be fast. However, during Icehouse the smoke tag was repurposed as a way to let neutron not backslide (so it's massively overloaded with network tests). It current runs at about 15 minutes on neutron jobs. This is why grenade neutron takes *so* long, because we run tempest smoke twice. The smoke tag needs a diet. I believe our working definition should be something as follows: - Total run time should be fast (= 5 minutes) - No negative tests - No admin tests - No tests that test optional extensions - No tests that test advanced services (like lbaas, vpnaas) - No proxy service tests The criteria for a good set of tests is CRUD operations on basic services. For instance, with compute we should have built a few servers, ensure we can shut them down. For neutron we should have done some basic network / port plugging. We also previously had the 'smoke' tag include all of the scenario tests, which was fine when we had 6 scenario tests. However as those have grown I think that should be trimmed back to a few basic through scenarios. The results of this are - https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:smoke,n,z The impacts on our upstream gate will mean that grenade jobs will speed up dramatically (20 minutes faster on grenade neutron). There is one edge condition which exists, which is the check-tempest-dsvm-neutron-icehouse job. Neutron couldn't pass either a full or parallel tempest run in icehouse (it's far too racy). So that's current running the smoke-serial tag. This would end up reducing the number of tests run on that job. However, based on the number of rechecks I've had to run in this series, that job is currently at about a 30% fail rate - http://goo.gl/N2w7qc - which means some test reduction is probably in order anyway, as it's mostly just preventing other people from landing unrelated patches. This was something we were originally planning on doing during the QA Sprint but ran out of time. It looks like we'll plan to land this right after Tempest 4 is cut this week, so that people that really want the old behavior can stay on the Tempest 4 release, but master is moving forward. I think that once we trim down we can decide to point add specific tests later. I expect smoke to be a bit more fluid over time, so it's not a tag that anyone should count on a test going into that tag and staying forever. -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
