[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

Since this bug is already marked as fix released for the cloud-archive, let's please track this issue in the following bug: https://bugs.launchpad.net/ubuntu/+bug/1744062 ** No longer affects: keepalived (Ubuntu) ** No longer affects: keepalived (Ubuntu Xenial) ** No longer affects: keepalived

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

Since this bug is already marked as fix released for the cloud-archive, let's please track this issue in the following bug: https://bugs.launchpad.net/ubuntu/+bug/1744062 ** No longer affects: keepalived (Ubuntu) ** No longer affects: keepalived (Ubuntu Xenial) ** No longer affects: keepalived

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

Xav, I just checked and the patch you referenced can be backported fairly cleanly to at least keepalived 1:1.2.19-1 (xenial/mitaka) and above. ** Changed in: keepalived (Ubuntu Xenial) Importance: Undecided => Critical ** Changed in: keepalived (Ubuntu Xenial) Status: New => Triaged

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

Xav, I just checked and the patch you referenced can be backported fairly cleanly to at least keepalived 1:1.2.19-1 (xenial/mitaka) and above. ** Changed in: keepalived (Ubuntu Xenial) Importance: Undecided => Critical ** Changed in: keepalived (Ubuntu Xenial) Status: New => Triaged

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

** No longer affects: keepalived (Ubuntu Zesty) -- You received this bug notification because you are a member of Ubuntu Server, which is subscribed to keepalived in Ubuntu. https://bugs.launchpad.net/bugs/1731595 Title: L3 HA: multiple agents are active at the same time To manage

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

** No longer affects: keepalived (Ubuntu Zesty) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1731595 Title: L3 HA: multiple agents are active at the same time To manage notifications about this

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

** Also affects: keepalived (Ubuntu) Importance: Undecided Status: New ** No longer affects: keepalived (Ubuntu Artful) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1731595 Title: L3

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

** Also affects: keepalived (Ubuntu) Importance: Undecided Status: New ** No longer affects: keepalived (Ubuntu Artful) -- You received this bug notification because you are a member of Ubuntu Server, which is subscribed to keepalived in Ubuntu. https://bugs.launchpad.net/bugs/1731595

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

Corey, as far as I'm aware there isn't a bug open for the keepalived package (for Xenial at least). Are you suggesting that we open a bug for a backport to the current cloudarchive package? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

Xav, do we have a bug open against the keepalived package for this? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1731595 Title: L3 HA: multiple agents are active at the same time To manage

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

Comment for the folks that are noticing this as 'fix released' but still affected - see https://github.com/acassen/keepalived/commit/e90a633c34fbe6ebbb891aa98bf29ce579b8b45c for the rest of this fix, we need keepalived to be at least 1.4.0 in order to have this commit. -- You received this bug

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

Hello On Xenial. a customer reported that when they upgraded neutron-openvswitch-agent, In some hosts's(not all of them), all ports are deleted after it tried restarting agent service. and all instances on the host lost network connectivity Could be race condition, some had issue but some

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

This bug was fixed in the package neutron - 2:8.4.0-0ubuntu6~cloud0 --- neutron (2:8.4.0-0ubuntu6~cloud0) trusty-mitaka; urgency=medium . * New update for the Ubuntu Cloud Archive. . neutron (2:8.4.0-0ubuntu6) xenial; urgency=medium . *

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

This bug was fixed in the package neutron - 2:8.4.0-0ubuntu6 --- neutron (2:8.4.0-0ubuntu6) xenial; urgency=medium * d/p/set-ha-network-port-to-down-when-l3-agent-starts.patch, d/p/call-update_all_ha_network_port_statuses-on-agent-start.patch: Cherry-pick from upstream

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

** Tags removed: neutron-proactive-backport-potential -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1731595 Title: L3 HA: multiple agents are active at the same time To manage notifications about

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

This bug was fixed in the package neutron - 2:9.4.1-0ubuntu1~cloud1 --- neutron (2:9.4.1-0ubuntu1~cloud1) xenial-newton; urgency=medium . * d/p/call-update_all_ha_network_port_statuses-on-agent-start.patch: Cherry-pick from upstream stable/ocata branch to prevent multiple

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

Regression testing was successful for xenial-mitaka-proposed and trusty- mitaka-proposed. xenial-mitaka-proposed with development charms: == Totals == Ran: 102 tests in 984.1188 sec. - Passed: 94 - Skipped: 8 - Expected Fail: 0 - Unexpected Success: 0 - Failed: 0 Sum of execute time

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

Regression testing was successful for xenial-newton-proposed. xenial-newton-proposed with development charms: == Totals == Ran: 102 tests in 1480.0247 sec. - Passed: 94 - Skipped: 8 - Expected Fail: 0 - Unexpected Success: 0 - Failed: 0 Sum of execute time for each test: 896.7256

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

This bug was fixed in the package neutron - 2:10.0.4-0ubuntu2~cloud0 --- neutron (2:10.0.4-0ubuntu2~cloud0) xenial-ocata; urgency=medium . * New update for the Ubuntu Cloud Archive. . neutron (2:10.0.4-0ubuntu2) zesty; urgency=medium . *

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

I wasn't able to change the upstream status back to "New" so I've opened a new bug to track this at https://bugs.launchpad.net/ubuntu/artful/+source/neutron/+bug/1744062. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

I think we need to get a new bug opened for this. As it's been marked fix released upstream it's probably not on anyone's radar. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1731595 Title: L3 HA:

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

As I am unable to reproduce it, I will be happy if someone takes over this issue. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1731595 Title: L3 HA: multiple agents are active at the same time To

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

It seems as if this bug surfaces due to load issues. While the fix provided by Venkata (https://review.openstack.org/#/c/522641/) should help clean things up at the time of l3 agent restart, issues seem to come back later down the line in some circumstances. xavpaice mentioned he saw multiple

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

Hi jhebden, thanks for the feedback. Yes, Pike has a newer pacakage versions. Once a release is GA, we stay at the major version (ie. 10 in the case of Ocata, 10.0.4) so as not to introduce any new features to a stable release. -- You received this bug notification because you are a member of

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

Hi, Same issue is coming for openstack pike with openstack-neutron-11.0.3 on Centos 7. Any solution/patch? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1731595 Title: L3 HA: multiple agents are

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

Hi Corey, Unfortunately, in the case of the cloud where we are seeing this behaviour, the updated package which Xav installed per his previous comment does not seem to have addressed the issue. This was neutron 10.0.4-0ubuntu1~cloud0 from Cloud Archive xenial-updates/ocata. I did notice that the

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

** Tags added: neutron-proactive-backport-potential -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1731595 Title: L3 HA: multiple agents are active at the same time To manage notifications about

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

Hi Xav, do you have any more feedback on ocata-proposed testing? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1731595 Title: L3 HA: multiple agents are active at the same time To manage

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

Hello Xav, or anyone else affected, Accepted neutron into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:8.4.0-0ubuntu6 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

This bug was fixed in the package neutron - 2:11.0.2-0ubuntu1.1~cloud0 --- neutron (2:11.0.2-0ubuntu1.1~cloud0) xenial-pike; urgency=medium . * New update for the Ubuntu Cloud Archive. . neutron (2:11.0.2-0ubuntu1.1) artful; urgency=medium . * d/gbp.conf: Set debian-branch

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

This bug was fixed in the package neutron - 2:11.0.2-0ubuntu1.1 --- neutron (2:11.0.2-0ubuntu1.1) artful; urgency=medium * d/gbp.conf: Set debian-branch to stable/pike. * New upstream version. * New stable point release for OpenStack Pike (LP: #1734990). *

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

I'm marking this as fix released for artful/pike as it has passed regression testing successfully and it is paired up with a stable point release that has CVE fixes. Xav, I know you are testing ocata still and it is up in the air still as to whether this has fixed your problem. Please keep us

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

s/fix released/verified -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1731595 Title: L3 HA: multiple agents are active at the same time To manage notifications about this bug go to:

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

We have installed the Ocata -proposed package, however the situation is this: - there's 464 routers configured, on 3 Neutron gateway hosts, using l3-ha, and each router is scheduled to all 3 hosts. - we installed the package because were in a situation with a current incident with multiple l3

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

Regression testing has completed successfully for artful, zesty, xenial- pike, and xenial-ocata. artful-pike-proposed with stable charms: == Totals == Ran: 102 tests in 1551.5835 sec. - Passed: 93 - Skipped: 9 - Expected Fail: 0 - Unexpected Success: 0 - Failed: 0 Sum of execute

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

I've uploaded new versions of neutron with backported patches to fix this issue to xenial (awaiting SRU team review) and newton-staging. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1731595 Title:

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

SRU addendum for mitaka (xenial) and newton. [Regression Potential] For mitaka (xenial) and newton, the regression potential is a little higher than that of ocata+ as the patch(es) aren't available on the corresponding upstream branches (they're EOL). These patches were cherry picked from the

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

Hi Xav, I took a look at the code and confirmed that this does appear to affect Newton and Mitaka so I've targeted those releases as well. We'll need to backport the Ocata patches to Newton and then Mitaka. Corey ** Also affects: neutron (Ubuntu Xenial) Importance: Undecided Status:

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

Please note, we now have a client affected by this running Mitaka as well. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1731595 Title: L3 HA: multiple agents are active at the same time To manage

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

** Changed in: cloud-archive/queens Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1731595 Title: L3 HA: multiple agents are active at the same time To

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

Hello Xav, or anyone else affected, Accepted neutron into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:10.0.4-0ubuntu2 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

SRU details for Ubuntu: [Impact] Details of the issue are described thoroughly in this bug report. The fix prevents multiple L3HA masters from existing at the same time, and is already upstream for all affected branches. [Test Case] The following SRU process will be followed:

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

Reviewed: https://review.openstack.org/522792 Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=385ac553e33f12c34e8a23459337b2f0af0b75eb Submitter: Zuul Branch:stable/ocata commit 385ac553e33f12c34e8a23459337b2f0af0b75eb Author: venkata anil

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

Hello Xav, or anyone else affected, Accepted neutron into pike-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository. Please help us by testing this new package. To enable the -proposed repository: sudo

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

Reviewed: https://review.openstack.org/522641 Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=9ed693228f90251c0f03fb842ef19628b439f9bc Submitter: Zuul Branch:master commit 9ed693228f90251c0f03fb842ef19628b439f9bc Author: venkata anil Date:

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

Hello Xav, or anyone else affected, Accepted neutron into artful-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:11.0.2-0ubuntu1.1 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

@jog ack, confirmed. We're tracking it as such. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1731595 Title: L3 HA: multiple agents are active at the same time To manage notifications about this

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

** Also affects: neutron (Ubuntu Bionic) Importance: High Assignee: Corey Bryant (corey.bryant) Status: Triaged ** Also affects: neutron (Ubuntu Zesty) Importance: Undecided Status: New ** Also affects: neutron (Ubuntu Artful) Importance: Undecided Status: New

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

This bug falls under the Canonical Cloud Engineering service-level agreement (SLA) process, as a field critical bug. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1731595 Title: L3 HA: multiple

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

** Changed in: cloud-archive Status: New => Triaged ** Changed in: cloud-archive Importance: Undecided => High ** Changed in: cloud-archive Assignee: (unassigned) => Corey Bryant (corey.bryant) ** Changed in: neutron (Ubuntu) Status: New => Triaged ** Changed in: neutron

[Bug 1731595] Re: L3 HA: multiple agents are active at the same time

** Also affects: neutron (Ubuntu) Importance: Undecided Status: New ** Also affects: cloud-archive Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.