On 3 February 2016 at 18:49, Armando M. <[email protected]> wrote: > > > On 3 February 2016 at 04:28, Sean Dague <[email protected]> wrote: > >> On 02/02/2016 10:03 PM, Matthew Treinish wrote: >> > On Tue, Feb 02, 2016 at 05:09:47PM -0800, Armando M. wrote: >> >> Folks, >> >> >> >> We have some IPv6 related bugs [1,2,3] that have been lingering for >> some >> >> time now. They have been hurting the gate (e.g. [4] the most recent >> >> offending failure) and since it looks like they have been without >> owners >> >> nor a plan of action for some time, I made the hard decision of >> skipping >> >> them [5] ahead of the busy times ahead. >> > >> > So TBH I don't think the failure rate for these tests are really at a >> point >> > necessitating a skip: >> > >> > >> http://status.openstack.org/openstack-health/#/test/tempest.scenario.test_network_v6.TestGettingAddress.test_multi_prefix_slaac >> > >> http://status.openstack.org/openstack-health/#/test/tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_dhcp6_stateless_from_os >> > >> http://status.openstack.org/openstack-health/#/test/tempest.scenario.test_network_v6.TestGettingAddress.test_dhcp6_stateless_from_os >> > >> > (also just a cool side-note, you can see the very obvious performance >> regression >> > caused by the keystonemiddleware release and when we excluded that >> version in >> > requirements) >> > >> > Well, test_dualnet_dhcp6_stateless_from_os is kinda there with a ~10% >> failure >> > rate, but the other 2 really aren't. I normally would be -1 on the skip >> patch >> > because of that. We try to save the skips for cases where the bugs are >> really >> > severe and preventing productivity at a large scale. >> > >> > But, in this case these ipv6 tests are kinda of out of place in >> tempest. Having >> > all the permutations of possible ip allocation configurations always >> seemed a >> > bit too heavy handed. These tests are also consistently in the top 10 >> slowest >> > for a run. We really should have trimmed down this set a while ago so >> we're only >> > have a single case in tempest. Neutron should own the other possible >> > configurations as an in-tree test. >> > >> > Brian Haley has a patch up from Dec. that was trying to clean it up: >> > >> > https://review.openstack.org/#/c/239868/ >> > >> > We probably should revisit that soon, since quite clearly no one is >> looking at >> > these right now. >> >> We definitely shouldn't be running all the IPv6 tests. >> >> But I also think the assumption that the failure rate is low is not a >> valid reason to keep a test. Unreliable tests that don't have anyone >> looking into them should be deleted. They are providing negative value. >> Because people just recheck past them even if their code made the race >> worse. So any legitimate issues they are exposing are being ignored. >> >> If the neutron PTL wants tests pulled, we should just do it. >> >> > Thanks for the support! Having said, I think it's important to make a > judgement call on a case by case basis, because removing tests blindly > might as well backfire. > > In this specific instance and all things considered, merging [2] (or even > better [1]) feel like a net gain. > > Cheers, > Armando > > [1] https://review.openstack.org/#/c/239868/ > [2] https://review.openstack.org/#/c/275457/ > >
Btw I did respin [1], because I am still seeing intermittent failures. [1] https://review.openstack.org/#/c/275457/ -Sean >> >> -- >> Sean Dague >> http://dague.net >> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> [email protected]?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
