Yes, this is all great Salvatore and Armando! Thank you for all of this work and the explanation behind it all.
Kyle On Dec 2, 2013, at 2:24 PM, Eugene Nikanorov <enikano...@mirantis.com> wrote: > Salvatore and Armando, thanks for your great work and detailed explanation! > > Eugene. > > > On Mon, Dec 2, 2013 at 11:48 PM, Joe Gordon <joe.gord...@gmail.com> wrote: > > On Dec 2, 2013 9:04 PM, "Salvatore Orlando" <sorla...@nicira.com> wrote: > > > > Hi, > > > > As you might have noticed, there has been some progress on parallel tests > > for neutron. > > In a nutshell: > > * Armando fixed the issue with IP address exhaustion on the public network > >  > > * Salvatore has now a patch which has a 50% success rate (the last failures > > are because of me playing with it)  > > * Salvatore is looking at putting back on track full isolation  > > * All the bugs affecting parallel tests can be queried here  > > * This blueprint tracks progress made towards enabling parallel testing  > > > > --------- > > The long story is as follows: > > Parallel testing basically is not working because parallelism means higher > > contention for public IP addresses. This was made worse by the fact that > > some tests created a router with a gateway set but never deleted it. As a > > result, there were even less addresses in the public range. > >  was already merged and with  we shall make the public network for > > neutron a /24 (the full tempest suite is still showing a lot of IP > > exhaustion errors). > > > > However, this was just one part of the issue. The biggest part actually > > lied with the OVS agent and its interactions with the ML2 plugin. A few > > patches (, , ) were already pushed to reduce the number of > > notifications sent from the plugin to the agent. However, the agent is > > organised in a way such that a notification is immediately acted upon thus > > preempting the main agent loop, which is the one responsible for wiring > > ports into networks. Considering the high level of notifications currently > > sent from the server, this becomes particularly wasteful if one consider > > that security membership updates for ports trigger global > > iptables-save/restore commands which are often executed in rapid > > succession, thus resulting in long delays for wiring VIFs to the > > appropriate network. > > With the patch  we are refactoring the agent to make it more efficient. > > This is not production code, but once we'll get close to 100% pass for > > parallel testing this patch will be split in several patches, properly > > structured, and hopefully easy to review. > > It is worth noting there is still work to do: in some cases the loop still > > takes too long, and it has been observed ovs commands taking even 10 > > seconds to complete. To this aim, it is worth considering use of async > > processes introduced in  as well as leveraging ovsdb monitoring  for > > limiting queries to ovs database. > > We're still unable to explain some failures where the network appears to be > > correctly wired (floating IP, router port, dhcp port, and VIF port), but > > the SSH connection fails. We're hoping to reproduce this failure patter > > locally. > > > > Finally, the tempest patch for full tempest isolation should be made usable > > soon. Having another experimental job for it is something worth considering > > as for some reason it is not always easy reproducing the same failure modes > > exhibited on the gate. > > > > Regards, > > Salvatore > > > > Awesome work, thanks for the update. > > > >  https://review.openstack.org/#/c/58054/ > >  https://review.openstack.org/#/c/57420/ > >  https://review.openstack.org/#/c/53459/ > >  https://review.openstack.org/#/c/58284/ > >  https://review.openstack.org/#/c/58860/ > >  https://review.openstack.org/#/c/58597/ > >  https://review.openstack.org/#/c/58415/ > >  https://review.openstack.org/#/c/45676/ > >  https://bugs.launchpad.net/neutron/+bug/1177973 > >  > > https://bugs.launchpad.net/neutron/+bugs?field.tag=neutron-parallel&field.tags_combinator=ANY > >  https://blueprints.launchpad.net/neutron/+spec/neutron-tempest-parallel > > > > _______________________________________________ > > OpenStack-dev mailing list > > OpenStackemail@example.com > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > _______________________________________________ > OpenStack-dev mailing list > OpenStackfirstname.lastname@example.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > _______________________________________________ > OpenStack-dev mailing list > OpenStackemail@example.com > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _______________________________________________ OpenStack-dev mailing list OpenStackfirstname.lastname@example.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev