Re: [openstack-dev] [Neutron] minimal scope covered by third-party testing
Hi Salvatore, On 03/04/2014 14:56, Salvatore Orlando wrote: Hi Simon, snip I hope stricter criteria will be enforced for Juno; I personally think every CI should run at least the smoketest suite for L2/L3 services (eg: load balancer scenario will stay optional). I had a little thinking about this and I feel like it might not have caught _immediately_ the issue Kyle talked about [1]. Let's rewind the time line: 1/ Change to *Nova* adding external events API is merged https://review.openstack.org/#/c/76388/ 2/ Change to *Neutron* notifying Nova when ports are ready is merged https://review.openstack.org/#/c/75253/ 3/ Change to *Nova* making libvirt wait for Neutron notifications is merged https://review.openstack.org/#/c/74832/ At this point and assuming that the external ODL CI system were running the L2/L3 smoke tests, change #3 could have passed since external Neutron CI aren't voting for Nova. Instead it would have voted against any subsequent change to Neutron. Simon [1] https://bugs.launchpad.net/neutron/+bug/1301449 Salvatore [1] https://review.openstack.org/#/c/75304/ On 3 April 2014 12:28, Simon Pasquier simon.pasqu...@bull.net mailto:simon.pasqu...@bull.net wrote: Hi, I'm looking at [1] but I see no requirement of which Tempest tests should be executed. In particular, I'm a bit puzzled that it is not mandatory to boot an instance and check that it gets connected to the network. To me, this is the very minimum for asserting that your plugin or driver is working with Neutron *and* Nova (I'm not even talking about security groups). I had a quick look at the existing 3rd party CI systems and I found none running this kind of check (correct me if I'm wrong). Thoughts? [1] https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers -- Simon Pasquier Software Engineer (OpenStack Expertise Center) Bull, Architect of an Open World Phone: + 33 4 76 29 71 49 tel:%2B%2033%204%2076%2029%2071%2049 http://www.bull.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] minimal scope covered by third-party testing
Hi Simon, You are absolutely right in your train of thoughts: unless the third-party CI monitors and vets all the potential changes it cares about there's always a chance something might break. This is why I think it's important that each Neutron third party CI should not only test Neutron changes, but also Nova's, DevStack's and Tempest's. Filters may be added to test only the relevant subtrees. For instance, the VMware CI runs the full suite of tempest smoke tests, as they come from upstream and it vets all the changes that go in Tempest made to API and scenario tests as well as configuration changes. As for Nova, we test changes to the vif parts, and for DevStack, we validate changes made to lib/neutron*. Vetting all the changes coming in VS only the ones that can potentially break third-party support is a balancing act when you don't have infinite resources at your disposal, or you're just ramping up the CI infrastructure. Cheers, Armando On 4 April 2014 02:00, Simon Pasquier simon.pasqu...@bull.net wrote: Hi Salvatore, On 03/04/2014 14:56, Salvatore Orlando wrote: Hi Simon, snip I hope stricter criteria will be enforced for Juno; I personally think every CI should run at least the smoketest suite for L2/L3 services (eg: load balancer scenario will stay optional). I had a little thinking about this and I feel like it might not have caught _immediately_ the issue Kyle talked about [1]. Let's rewind the time line: 1/ Change to *Nova* adding external events API is merged https://review.openstack.org/#/c/76388/ 2/ Change to *Neutron* notifying Nova when ports are ready is merged https://review.openstack.org/#/c/75253/ 3/ Change to *Nova* making libvirt wait for Neutron notifications is merged https://review.openstack.org/#/c/74832/ At this point and assuming that the external ODL CI system were running the L2/L3 smoke tests, change #3 could have passed since external Neutron CI aren't voting for Nova. Instead it would have voted against any subsequent change to Neutron. Simon [1] https://bugs.launchpad.net/neutron/+bug/1301449 Salvatore [1] https://review.openstack.org/#/c/75304/ On 3 April 2014 12:28, Simon Pasquier simon.pasqu...@bull.net mailto:simon.pasqu...@bull.net wrote: Hi, I'm looking at [1] but I see no requirement of which Tempest tests should be executed. In particular, I'm a bit puzzled that it is not mandatory to boot an instance and check that it gets connected to the network. To me, this is the very minimum for asserting that your plugin or driver is working with Neutron *and* Nova (I'm not even talking about security groups). I had a quick look at the existing 3rd party CI systems and I found none running this kind of check (correct me if I'm wrong). Thoughts? [1] https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers -- Simon Pasquier Software Engineer (OpenStack Expertise Center) Bull, Architect of an Open World Phone: + 33 4 76 29 71 49 tel:%2B%2033%204%2076%2029%2071%2049 http://www.bull.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Neutron] minimal scope covered by third-party testing
Hi, I'm looking at [1] but I see no requirement of which Tempest tests should be executed. In particular, I'm a bit puzzled that it is not mandatory to boot an instance and check that it gets connected to the network. To me, this is the very minimum for asserting that your plugin or driver is working with Neutron *and* Nova (I'm not even talking about security groups). I had a quick look at the existing 3rd party CI systems and I found none running this kind of check (correct me if I'm wrong). Thoughts? [1] https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers -- Simon Pasquier Software Engineer (OpenStack Expertise Center) Bull, Architect of an Open World Phone: + 33 4 76 29 71 49 http://www.bull.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] minimal scope covered by third-party testing
Hi Simon, I agree with your concern. Let me point out however that VMware mine sweeper runs almost all the smoke suite. It's been down a few days for an internal software upgrade, so perhaps you have not seen any recent report from it. I've seen some CI systems testing as little as tempest.api.network. Since a criterion on the minimum set of tests to run was not defined prior to the release cycle, it was also not ok to enforce it once the system went live. The only thing active at the moment is a sort of purpose built lie detector [1]. I hope stricter criteria will be enforced for Juno; I personally think every CI should run at least the smoketest suite for L2/L3 services (eg: load balancer scenario will stay optional). Salvatore [1] https://review.openstack.org/#/c/75304/ On 3 April 2014 12:28, Simon Pasquier simon.pasqu...@bull.net wrote: Hi, I'm looking at [1] but I see no requirement of which Tempest tests should be executed. In particular, I'm a bit puzzled that it is not mandatory to boot an instance and check that it gets connected to the network. To me, this is the very minimum for asserting that your plugin or driver is working with Neutron *and* Nova (I'm not even talking about security groups). I had a quick look at the existing 3rd party CI systems and I found none running this kind of check (correct me if I'm wrong). Thoughts? [1] https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers -- Simon Pasquier Software Engineer (OpenStack Expertise Center) Bull, Architect of an Open World Phone: + 33 4 76 29 71 49 http://www.bull.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] minimal scope covered by third-party testing
I agree 100% on this in fact. One of the other concerns I have with the existing 3rd party CI systems is that, other than the audit review Salvatore mentions, who is ensuring they continue to run ok? Once they've been given voting rights, is anyone auditing these to ensure they continue to function ok? I suspect also that Simon is referring to the ODL ML2 MechanismDriver, which was broken with this commit [1] pushed in at the very end of Icehouse, and in fact is still broken unless you use the wonky workaround of telling Nova that VIF plugging isn't fatal and give it a timeout to wait. Better CI for ODL would have caught this, but I'm still somewhat saddened this was merged so late because now ODL is broken by default and the work to fix this is turning out to be more challenging than initially thought. :( Thanks, Kyle [1] https://review.openstack.org/#/c/75253/ On Thu, Apr 3, 2014 at 7:56 AM, Salvatore Orlando sorla...@nicira.com wrote: Hi Simon, I agree with your concern. Let me point out however that VMware mine sweeper runs almost all the smoke suite. It's been down a few days for an internal software upgrade, so perhaps you have not seen any recent report from it. I've seen some CI systems testing as little as tempest.api.network. Since a criterion on the minimum set of tests to run was not defined prior to the release cycle, it was also not ok to enforce it once the system went live. The only thing active at the moment is a sort of purpose built lie detector [1]. I hope stricter criteria will be enforced for Juno; I personally think every CI should run at least the smoketest suite for L2/L3 services (eg: load balancer scenario will stay optional). Salvatore [1] https://review.openstack.org/#/c/75304/ On 3 April 2014 12:28, Simon Pasquier simon.pasqu...@bull.net wrote: Hi, I'm looking at [1] but I see no requirement of which Tempest tests should be executed. In particular, I'm a bit puzzled that it is not mandatory to boot an instance and check that it gets connected to the network. To me, this is the very minimum for asserting that your plugin or driver is working with Neutron *and* Nova (I'm not even talking about security groups). I had a quick look at the existing 3rd party CI systems and I found none running this kind of check (correct me if I'm wrong). Thoughts? [1] https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers -- Simon Pasquier Software Engineer (OpenStack Expertise Center) Bull, Architect of an Open World Phone: + 33 4 76 29 71 49 http://www.bull.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] minimal scope covered by third-party testing
Thanks Salvatore and Kyle for your feedback. Kyle, you're right, my question has been kicked off by the ML2 ODL bug. I didn't want to point fingers but rather understand the mid/long-term plan for 3rd party testing. I'm happy to see that this is taken into account and hopefully the Juno cycle will provide time to implement the correct level of testing. Regards, Simon On 03/04/2014 15:26, Kyle Mestery wrote: I agree 100% on this in fact. One of the other concerns I have with the existing 3rd party CI systems is that, other than the audit review Salvatore mentions, who is ensuring they continue to run ok? Once they've been given voting rights, is anyone auditing these to ensure they continue to function ok? I suspect also that Simon is referring to the ODL ML2 MechanismDriver, which was broken with this commit [1] pushed in at the very end of Icehouse, and in fact is still broken unless you use the wonky workaround of telling Nova that VIF plugging isn't fatal and give it a timeout to wait. Better CI for ODL would have caught this, but I'm still somewhat saddened this was merged so late because now ODL is broken by default and the work to fix this is turning out to be more challenging than initially thought. :( Thanks, Kyle [1] https://review.openstack.org/#/c/75253/ On Thu, Apr 3, 2014 at 7:56 AM, Salvatore Orlando sorla...@nicira.com wrote: Hi Simon, I agree with your concern. Let me point out however that VMware mine sweeper runs almost all the smoke suite. It's been down a few days for an internal software upgrade, so perhaps you have not seen any recent report from it. I've seen some CI systems testing as little as tempest.api.network. Since a criterion on the minimum set of tests to run was not defined prior to the release cycle, it was also not ok to enforce it once the system went live. The only thing active at the moment is a sort of purpose built lie detector [1]. I hope stricter criteria will be enforced for Juno; I personally think every CI should run at least the smoketest suite for L2/L3 services (eg: load balancer scenario will stay optional). Salvatore [1] https://review.openstack.org/#/c/75304/ On 3 April 2014 12:28, Simon Pasquier simon.pasqu...@bull.net wrote: Hi, I'm looking at [1] but I see no requirement of which Tempest tests should be executed. In particular, I'm a bit puzzled that it is not mandatory to boot an instance and check that it gets connected to the network. To me, this is the very minimum for asserting that your plugin or driver is working with Neutron *and* Nova (I'm not even talking about security groups). I had a quick look at the existing 3rd party CI systems and I found none running this kind of check (correct me if I'm wrong). Thoughts? [1] https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers -- Simon Pasquier Software Engineer (OpenStack Expertise Center) Bull, Architect of an Open World Phone: + 33 4 76 29 71 49 http://www.bull.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
