Salvatore and Armando, thanks for your great work and detailed explanation!
Eugene. On Mon, Dec 2, 2013 at 11:48 PM, Joe Gordon <joe.gord...@gmail.com> wrote: > > On Dec 2, 2013 9:04 PM, "Salvatore Orlando" <sorla...@nicira.com> wrote: > > > > Hi, > > > > As you might have noticed, there has been some progress on parallel > tests for neutron. > > In a nutshell: > > * Armando fixed the issue with IP address exhaustion on the public > network [1] > > * Salvatore has now a patch which has a 50% success rate (the last > failures are because of me playing with it) [2] > > * Salvatore is looking at putting back on track full isolation [3] > > * All the bugs affecting parallel tests can be queried here [10] > > * This blueprint tracks progress made towards enabling parallel testing > [11] > > > > --------- > > The long story is as follows: > > Parallel testing basically is not working because parallelism means > higher contention for public IP addresses. This was made worse by the fact > that some tests created a router with a gateway set but never deleted it. > As a result, there were even less addresses in the public range. > > [1] was already merged and with [4] we shall make the public network for > neutron a /24 (the full tempest suite is still showing a lot of IP > exhaustion errors). > > > > However, this was just one part of the issue. The biggest part actually > lied with the OVS agent and its interactions with the ML2 plugin. A few > patches ([5], [6], [7]) were already pushed to reduce the number of > notifications sent from the plugin to the agent. However, the agent is > organised in a way such that a notification is immediately acted upon thus > preempting the main agent loop, which is the one responsible for wiring > ports into networks. Considering the high level of notifications currently > sent from the server, this becomes particularly wasteful if one consider > that security membership updates for ports trigger global > iptables-save/restore commands which are often executed in rapid > succession, thus resulting in long delays for wiring VIFs to the > appropriate network. > > With the patch [2] we are refactoring the agent to make it more > efficient. This is not production code, but once we'll get close to 100% > pass for parallel testing this patch will be split in several patches, > properly structured, and hopefully easy to review. > > It is worth noting there is still work to do: in some cases the loop > still takes too long, and it has been observed ovs commands taking even 10 > seconds to complete. To this aim, it is worth considering use of async > processes introduced in [8] as well as leveraging ovsdb monitoring [9] for > limiting queries to ovs database. > > We're still unable to explain some failures where the network appears to > be correctly wired (floating IP, router port, dhcp port, and VIF port), but > the SSH connection fails. We're hoping to reproduce this failure patter > locally. > > > > Finally, the tempest patch for full tempest isolation should be made > usable soon. Having another experimental job for it is something worth > considering as for some reason it is not always easy reproducing the same > failure modes exhibited on the gate. > > > > Regards, > > Salvatore > > > > Awesome work, thanks for the update. > > > [1] https://review.openstack.org/#/c/58054/ > > [2] https://review.openstack.org/#/c/57420/ > > [3] https://review.openstack.org/#/c/53459/ > > [4] https://review.openstack.org/#/c/58284/ > > [5] https://review.openstack.org/#/c/58860/ > > [6] https://review.openstack.org/#/c/58597/ > > [7] https://review.openstack.org/#/c/58415/ > > [8] https://review.openstack.org/#/c/45676/ > > [9] https://bugs.launchpad.net/neutron/+bug/1177973 > > [10] > https://bugs.launchpad.net/neutron/+bugs?field.tag=neutron-parallel&field.tags_combinator=ANY > > [11] > https://blueprints.launchpad.net/neutron/+spec/neutron-tempest-parallel > > > > _______________________________________________ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev