Re: [openstack-dev] [nova][qa] proposal for moving forward on cells/tempest testing
On 7/15/2014 12:36 AM, Sean Dague wrote: On 07/14/2014 07:44 PM, Matt Riedemann wrote: Today we only gate on exercises in devstack for cells testing coverage in the gate-devstack-dsvm-cells job. The cells tempest non-voting job was moving to the experimental queue here [1] since it doesn't work with a lot of the compute API tests. I think we all agreed to tar and feather comstud if he didn't get Tempest working (read: passing) with cells enabled in Juno. The first part of this is just figuring out where we sit with what's failing in Tempest (in the check-tempest-dsvm-cells-full job). I'd like to propose that we do the following to get the ball rolling: 1. Add an option to tempest.conf under the compute-feature-enabled section to toggle cells and then use that option to skip tests that we know will fail in cells, e.g. security group tests. I don't think we should do that. Part of creating the feature matrix in devstack gate included the follow on idea of doing extension selection based on branch or feature. I'm happy if that gets finished, then tests are skipped by known not working extensions, but just landing a ton of tempest ifdefs that will all be removed is feeling very gorpy. Especially as we're now at Juno 2, which was supposed to be the checkpoint for this being on track for completion and... people are just talking about starting. 2. Open bugs for all of the tests we're skipping so we can track closing those down, assuming they aren't already reported. [2] 3. Once the known failures are being skipped, we can move check-tempest-dsvm-cells-full out of the experimental queue. I'm not proposing that it'd be voting right away, I think we have to see it burn in for awhile first. With at least this plan we should be able to move forward on identifying issues and getting some idea for how much of Tempest doesn't work with cells and the effort involved in making it work. Thoughts? If there aren't any objections, I said I'd work on the qa-spec and can start doing the grunt-work of opening bugs and skipping tests. [1] https://review.openstack.org/#/c/87982/ [2] https://bugs.launchpad.net/nova/+bugs?field.tag=cells+ All the rest is fine, I just think we should work on the proper way to skip things. -Sean ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev OK I don't know anything about the extensions in devstack-gate or how the skips would work then, I'll have to bug some people in IRC unless there is an easy example that can be pointed out here. -- Thanks, Matt Riedemann ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [nova][qa] proposal for moving forward on cells/tempest testing
Today we only gate on exercises in devstack for cells testing coverage in the gate-devstack-dsvm-cells job. The cells tempest non-voting job was moving to the experimental queue here [1] since it doesn't work with a lot of the compute API tests. I think we all agreed to tar and feather comstud if he didn't get Tempest working (read: passing) with cells enabled in Juno. The first part of this is just figuring out where we sit with what's failing in Tempest (in the check-tempest-dsvm-cells-full job). I'd like to propose that we do the following to get the ball rolling: 1. Add an option to tempest.conf under the compute-feature-enabled section to toggle cells and then use that option to skip tests that we know will fail in cells, e.g. security group tests. 2. Open bugs for all of the tests we're skipping so we can track closing those down, assuming they aren't already reported. [2] 3. Once the known failures are being skipped, we can move check-tempest-dsvm-cells-full out of the experimental queue. I'm not proposing that it'd be voting right away, I think we have to see it burn in for awhile first. With at least this plan we should be able to move forward on identifying issues and getting some idea for how much of Tempest doesn't work with cells and the effort involved in making it work. Thoughts? If there aren't any objections, I said I'd work on the qa-spec and can start doing the grunt-work of opening bugs and skipping tests. [1] https://review.openstack.org/#/c/87982/ [2] https://bugs.launchpad.net/nova/+bugs?field.tag=cells+ -- Thanks, Matt Riedemann ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][qa] proposal for moving forward on cells/tempest testing
On Jul 14, 2014, at 10:44 AM, Matt Riedemann mrie...@linux.vnet.ibm.com wrote: Today we only gate on exercises in devstack for cells testing coverage in the gate-devstack-dsvm-cells job. The cells tempest non-voting job was moving to the experimental queue here [1] since it doesn't work with a lot of the compute API tests. I think we all agreed to tar and feather comstud if he didn't get Tempest working (read: passing) with cells enabled in Juno. The first part of this is just figuring out where we sit with what's failing in Tempest (in the check-tempest-dsvm-cells-full job). I'd like to propose that we do the following to get the ball rolling: 1. Add an option to tempest.conf under the compute-feature-enabled section to toggle cells and then use that option to skip tests that we know will fail in cells, e.g. security group tests. I think I was told tempest could infer cells from devstack config or something? I dunno the right way to do this. But, I'm basically +1 to all 3 of these. I think we just skip the broken tests for now and iterate on unskipping things one by one. - Chris 2. Open bugs for all of the tests we're skipping so we can track closing those down, assuming they aren't already reported. [2] 3. Once the known failures are being skipped, we can move check-tempest-dsvm-cells-full out of the experimental queue. I'm not proposing that it'd be voting right away, I think we have to see it burn in for awhile first. With at least this plan we should be able to move forward on identifying issues and getting some idea for how much of Tempest doesn't work with cells and the effort involved in making it work. Thoughts? If there aren't any objections, I said I'd work on the qa-spec and can start doing the grunt-work of opening bugs and skipping tests. [1] https://review.openstack.org/#/c/87982/ [2] https://bugs.launchpad.net/nova/+bugs?field.tag=cells+ -- Thanks, Matt Riedemann ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][qa] proposal for moving forward on cells/tempest testing
On 07/14/2014 07:44 PM, Matt Riedemann wrote: Today we only gate on exercises in devstack for cells testing coverage in the gate-devstack-dsvm-cells job. The cells tempest non-voting job was moving to the experimental queue here [1] since it doesn't work with a lot of the compute API tests. I think we all agreed to tar and feather comstud if he didn't get Tempest working (read: passing) with cells enabled in Juno. The first part of this is just figuring out where we sit with what's failing in Tempest (in the check-tempest-dsvm-cells-full job). I'd like to propose that we do the following to get the ball rolling: 1. Add an option to tempest.conf under the compute-feature-enabled section to toggle cells and then use that option to skip tests that we know will fail in cells, e.g. security group tests. I don't think we should do that. Part of creating the feature matrix in devstack gate included the follow on idea of doing extension selection based on branch or feature. I'm happy if that gets finished, then tests are skipped by known not working extensions, but just landing a ton of tempest ifdefs that will all be removed is feeling very gorpy. Especially as we're now at Juno 2, which was supposed to be the checkpoint for this being on track for completion and... people are just talking about starting. 2. Open bugs for all of the tests we're skipping so we can track closing those down, assuming they aren't already reported. [2] 3. Once the known failures are being skipped, we can move check-tempest-dsvm-cells-full out of the experimental queue. I'm not proposing that it'd be voting right away, I think we have to see it burn in for awhile first. With at least this plan we should be able to move forward on identifying issues and getting some idea for how much of Tempest doesn't work with cells and the effort involved in making it work. Thoughts? If there aren't any objections, I said I'd work on the qa-spec and can start doing the grunt-work of opening bugs and skipping tests. [1] https://review.openstack.org/#/c/87982/ [2] https://bugs.launchpad.net/nova/+bugs?field.tag=cells+ All the rest is fine, I just think we should work on the proper way to skip things. -Sean -- Sean Dague http://dague.net signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
