[openstack-dev] [qa] [heat] [neutron] - Status of Heat and Neutron tempest blueprints?
We're at Freeze, so I want to pick up and understand where we currently stand with both Neutron and Heat actually getting tested fully in the gate. First Neutron - https://blueprints.launchpad.net/tempest/+spec/fix-gate-tempest-devstack-vm-quantum-full We know that this is *close* as the full job is running non voting everywhere, and typically passing. How close are we? Or should we be defering this until Juno (which would be unfortunate). Second Heat - https://blueprints.launchpad.net/tempest/+spec/tempest-heat-integration The Heat tests that are in a normal Tempest job are relatively trivial surface verification, and in no way actually make sure that Heat is operating at a real level. This fact is a contributing factor to why Heat was broken in i2. The first real verification for Heat is in the Heat slow job (which we created to give Heat a separate time budget, because doing work that requires real guests takes time). The heat slow job looks like it is finally passing much of the time - http://logstash.openstack.org/#eyJzZWFyY2giOiIobWVzc2FnZTpcIkZpbmlzaGVkOiBTVUNDRVNTXCIgT1IgbWVzc2FnZTpcIkZpbmlzaGVkOiBGQUlMVVJFXCIpIEFORCBidWlsZF9uYW1lOmNoZWNrLXRlbXBlc3QtZHN2bS1uZXV0cm9uLWhlYXQtc2xvdyIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM5NDEwOTg3NDQ4OH0= It's seeing a 78% pass rate in check. Can anyone in the Heat team confirm that the Failures in this job are actually real failures on patches that should have been blocked? I'd like to get that turned on (and on all the projects) as soon as the Heat team is confident on it so that Heat actually participates in the tempest/devstack gate in a material way and we can prevent future issues where a keystone, nova, neutron or whatever change would break Heat in git. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] [heat] [neutron] - Status of Heat and Neutron tempest blueprints?
On Thu, Mar 06, 2014 at 07:53:03AM -0500, Sean Dague wrote: We're at Freeze, so I want to pick up and understand where we currently stand with both Neutron and Heat actually getting tested fully in the gate. First Neutron - https://blueprints.launchpad.net/tempest/+spec/fix-gate-tempest-devstack-vm-quantum-full We know that this is *close* as the full job is running non voting everywhere, and typically passing. How close are we? Or should we be defering this until Juno (which would be unfortunate). Can you please clarify - does FF apply to tempest *tests*? My assumption was that we could move from feature development to testing during the FF/RC phase of the cycle, the natural by-product of which will be additional tempest testcases.. Second Heat - https://blueprints.launchpad.net/tempest/+spec/tempest-heat-integration The Heat tests that are in a normal Tempest job are relatively trivial surface verification, and in no way actually make sure that Heat is operating at a real level. This fact is a contributing factor to why Heat was broken in i2. Yeah, we have a list of wishlist bugs so we can track the additional less-trivial tests we'd like to implement: https://bugs.launchpad.net/heat/+bugs?field.tag=tempest I was hoping we'd get patches at least proposed for most of these before Icehouse ships, but we'll have to see how it goes.. :) The first real verification for Heat is in the Heat slow job (which we created to give Heat a separate time budget, because doing work that requires real guests takes time). The heat slow job looks like it is finally passing much of the time - http://logstash.openstack.org/#eyJzZWFyY2giOiIobWVzc2FnZTpcIkZpbmlzaGVkOiBTVUNDRVNTXCIgT1IgbWVzc2FnZTpcIkZpbmlzaGVkOiBGQUlMVVJFXCIpIEFORCBidWlsZF9uYW1lOmNoZWNrLXRlbXBlc3QtZHN2bS1uZXV0cm9uLWhlYXQtc2xvdyIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM5NDEwOTg3NDQ4OH0= It's seeing a 78% pass rate in check. Can anyone in the Heat team confirm that the Failures in this job are actually real failures on patches that should have been blocked? I'd like to get that turned on (and on all the projects) as soon as the Heat team is confident on it so that Heat actually participates in the tempest/devstack gate in a material way and we can prevent future issues where a keystone, nova, neutron or whatever change would break Heat in git. Steve Baker can make the final decision, but I am +2 on turning it on - during the process of getting my instance-users patches through the gate, this test consistently found real issues with no false positives (other than those caused by other known bugs). Steve ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] [heat] [neutron] - Status of Heat and Neutron tempest blueprints?
On 03/06/2014 08:11 AM, Steven Hardy wrote: On Thu, Mar 06, 2014 at 07:53:03AM -0500, Sean Dague wrote: We're at Freeze, so I want to pick up and understand where we currently stand with both Neutron and Heat actually getting tested fully in the gate. First Neutron - https://blueprints.launchpad.net/tempest/+spec/fix-gate-tempest-devstack-vm-quantum-full We know that this is *close* as the full job is running non voting everywhere, and typically passing. How close are we? Or should we be defering this until Juno (which would be unfortunate). Can you please clarify - does FF apply to tempest *tests*? My assumption was that we could move from feature development to testing during the FF/RC phase of the cycle, the natural by-product of which will be additional tempest testcases.. It doesn't. Typically we're adding additional validation up to the release to catch things we believe are still exposed (especially in conjunction with bugs that are found). That being said, if the core teams here are punting, it would be good to know, so we don't waste time asking the questions over and over again. :) Second Heat - https://blueprints.launchpad.net/tempest/+spec/tempest-heat-integration The Heat tests that are in a normal Tempest job are relatively trivial surface verification, and in no way actually make sure that Heat is operating at a real level. This fact is a contributing factor to why Heat was broken in i2. Yeah, we have a list of wishlist bugs so we can track the additional less-trivial tests we'd like to implement: https://bugs.launchpad.net/heat/+bugs?field.tag=tempest I was hoping we'd get patches at least proposed for most of these before Icehouse ships, but we'll have to see how it goes.. :) +1. The more the better. The first real verification for Heat is in the Heat slow job (which we created to give Heat a separate time budget, because doing work that requires real guests takes time). The heat slow job looks like it is finally passing much of the time - http://logstash.openstack.org/#eyJzZWFyY2giOiIobWVzc2FnZTpcIkZpbmlzaGVkOiBTVUNDRVNTXCIgT1IgbWVzc2FnZTpcIkZpbmlzaGVkOiBGQUlMVVJFXCIpIEFORCBidWlsZF9uYW1lOmNoZWNrLXRlbXBlc3QtZHN2bS1uZXV0cm9uLWhlYXQtc2xvdyIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM5NDEwOTg3NDQ4OH0= It's seeing a 78% pass rate in check. Can anyone in the Heat team confirm that the Failures in this job are actually real failures on patches that should have been blocked? I'd like to get that turned on (and on all the projects) as soon as the Heat team is confident on it so that Heat actually participates in the tempest/devstack gate in a material way and we can prevent future issues where a keystone, nova, neutron or whatever change would break Heat in git. Steve Baker can make the final decision, but I am +2 on turning it on - during the process of getting my instance-users patches through the gate, this test consistently found real issues with no false positives (other than those caused by other known bugs). Great. Looking forward to turning that on. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] [heat] [neutron] - Status of Heat and Neutron tempest blueprints?
On 07/03/14 01:53, Sean Dague wrote: We're at Freeze, so I want to pick up and understand where we currently stand with both Neutron and Heat actually getting tested fully in the gate. First Neutron - https://blueprints.launchpad.net/tempest/+spec/fix-gate-tempest-devstack-vm-quantum-full We know that this is *close* as the full job is running non voting everywhere, and typically passing. How close are we? Or should we be defering this until Juno (which would be unfortunate). Second Heat - https://blueprints.launchpad.net/tempest/+spec/tempest-heat-integration The Heat tests that are in a normal Tempest job are relatively trivial surface verification, and in no way actually make sure that Heat is operating at a real level. This fact is a contributing factor to why Heat was broken in i2. The first real verification for Heat is in the Heat slow job (which we created to give Heat a separate time budget, because doing work that requires real guests takes time). The heat slow job looks like it is finally passing much of the time - http://logstash.openstack.org/#eyJzZWFyY2giOiIobWVzc2FnZTpcIkZpbmlzaGVkOiBTVUNDRVNTXCIgT1IgbWVzc2FnZTpcIkZpbmlzaGVkOiBGQUlMVVJFXCIpIEFORCBidWlsZF9uYW1lOmNoZWNrLXRlbXBlc3QtZHN2bS1uZXV0cm9uLWhlYXQtc2xvdyIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM5NDEwOTg3NDQ4OH0= It's seeing a 78% pass rate in check. Can anyone in the Heat team confirm that the Failures in this job are actually real failures on patches that should have been blocked? I'd like to get that turned on (and on all the projects) as soon as the Heat team is confident on it so that Heat actually participates in the tempest/devstack gate in a material way and we can prevent future issues where a keystone, nova, neutron or whatever change would break Heat in git. I've raised https://bugs.launchpad.net/tempest/+bug/1288970 to track the most common error ( NeutronResourcesTestJSON f ailed to reach CREATE_COMPLETE status within the required time (300 s)) tl;dr sometimes booting alone is taking 244s, so I think this test would be significantly more reliable if the timeout was raised. I would propose raising the default orchestration build_timeout to 600s for now, but it may need to go up to 1200s when the autoscaling scenario is enabled again. https://review.openstack.org/#/c/78756/ Once this change is in heat-slow should be reliable enough to make it voting. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
