Hi, can we should change bug status from Fix Released to Confirmed since
the fix was reverted, the problem behind it (lp bug 1916024) still
exist.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1927868
** Tags removed: neutron-proactive-backport-potential
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1927868
Title:
vRouter not working after update to 16.3.1
To manage notifications about this bug
** Tags added: neutron-proactive-backport-potential
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1927868
Title:
vRouter not working after update to 16.3.1
To manage notifications about this bug
** Merge proposal linked:
https://code.launchpad.net/~hopem/ubuntu/+source/neutron/+git/neutron/+merge/410053
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1927868
Title:
vRouter not working
Reviewed: https://review.opendev.org/c/openstack/neutron/+/809382
Committed:
https://opendev.org/openstack/neutron/commit/5049a8faf103324add24a09c74b9f41aba37ec75
Submitter: "Zuul (22348)"
Branch:stable/victoria
commit 5049a8faf103324add24a09c74b9f41aba37ec75
Author: Edward Hope-Morley
Reviewed: https://review.opendev.org/c/openstack/neutron/+/809384
Committed:
https://opendev.org/openstack/neutron/commit/f54658c203e66355e94fdd48fbd57682577fc928
Submitter: "Zuul (22348)"
Branch:stable/train
commit f54658c203e66355e94fdd48fbd57682577fc928
Author: Edward Hope-Morley
Date:
Reviewed: https://review.opendev.org/c/openstack/neutron/+/809219
Committed:
https://opendev.org/openstack/neutron/commit/b3a70fe75315fba061ecb7d6ac1d50a04768ec13
Submitter: "Zuul (22348)"
Branch:stable/wallaby
commit b3a70fe75315fba061ecb7d6ac1d50a04768ec13
Author: Edward Hope-Morley
Reviewed: https://review.opendev.org/c/openstack/neutron/+/805366
Committed:
https://opendev.org/openstack/neutron/commit/344fc0c8d2ce7d942606c834a54cb81f0b47aa37
Submitter: "Zuul (22348)"
Branch:master
commit 344fc0c8d2ce7d942606c834a54cb81f0b47aa37
Author: Edward Hope-Morley
Date: Fri
@hopem thanks for your nice reply and the complete overview of the
situation.
I do understand the issue with exception handling and propagation between
privsep and the reader.
As one cannot catch all exceptions or erroneous conditions that systems might
reach, a major improvement would be to
** Changed in: neutron
Assignee: (unassigned) => Edward Hope-Morley (hopem)
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1927868
Title:
vRouter not working after update to 16.3.1
To manage
Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/805366
** Changed in: neutron
Status: New => In Progress
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
@christian-rohmann The problem essentially boils down to the exception
at [1] being raised because prior to that [2] gets called as a result of
a timeout exception but the code is not actually catching the exception.
This was traced to be the result of a privileged call being used as
argument to
This bug was fixed in the package neutron - 2:15.3.4-0ubuntu1~cloud1
---
neutron (2:15.3.4-0ubuntu1~cloud1) bionic-train; urgency=medium
.
* d/p/revert-l3-ha-retry-when-setting-ha-router-gw-status.patch: Revert
upstream patch that introduced regression that prevented full
This bug was fixed in the package neutron - 2:17.2.0-0ubuntu1~cloud1
---
neutron (2:17.2.0-0ubuntu1~cloud1) focal-victoria; urgency=medium
.
* d/p/revert-l3-ha-retry-when-setting-ha-router-gw-status.patch: Revert
upstream patch that introduced regression that prevented full
This bug was fixed in the package neutron - 2:16.4.0-0ubuntu3
---
neutron (2:16.4.0-0ubuntu3) focal; urgency=medium
* d/p/revert-l3-ha-retry-when-setting-ha-router-gw-status.patch: Revert
upstream patch that introduced regression that prevented full restore
of HA routers on
This bug was fixed in the package neutron - 2:18.1.0-0ubuntu2
---
neutron (2:18.1.0-0ubuntu2) hirsute; urgency=medium
* d/p/revert-l3-ha-retry-when-setting-ha-router-gw-status.patch: Revert
upstream patch that introduced regression that prevented full restore
of HA routers
** Tags removed: verification-focal-done verification-hirsute-done
** Tags added: verification-done-focal verification-done-hirsute
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1927868
Title:
This bug was fixed in the package neutron - 2:18.1.0-0ubuntu2~cloud0
---
neutron (2:18.1.0-0ubuntu2~cloud0) focal-wallaby; urgency=medium
.
* New update for the Ubuntu Cloud Archive.
.
neutron (2:18.1.0-0ubuntu2) hirsute; urgency=medium
.
*
Verified bionic-ussuri/proposed and bionic-ussuri/train proposed using
[Test Case]
** Tags removed: verification-train-needed
** Tags added: verification-bionic-done verification-train-done
** Tags removed: verification-needed
** Tags added: verification-done
--
You received this bug
Verified hirtsue/proposed, focal-ussuri/proposed, focal-
victoria/proposed, focal-wallaby/proposed using [Test Case]
** Tags removed: verification-needed-focal verification-victoria-needed
verification-wallaby-needed
** Tags added: verification-focal-done verification-victoria-done
** Tags removed: verification-needed-hirsute
** Tags added: verification-hirsute-done
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1927868
Title:
vRouter not working after update to 16.3.1
To
This bug was fixed in the package neutron - 2:16.4.0-0ubuntu3~cloud0
---
neutron (2:16.4.0-0ubuntu3~cloud0) bionic-ussuri; urgency=medium
.
* New update for the Ubuntu Cloud Archive.
.
neutron (2:16.4.0-0ubuntu3) focal; urgency=medium
.
*
Verified bionic-ussuri/proposed using [Test Case]
** Tags removed: verification-ussuri-needed
** Tags added: verification-ussuri-done
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1927868
Title:
This bug was fixed in the package neutron -
2:18.1.0+git2021072117.147830620f-0ubuntu2~cloud0
---
neutron (2:18.1.0+git2021072117.147830620f-0ubuntu2~cloud0) focal-xena;
urgency=medium
.
* New update for the Ubuntu Cloud Archive.
.
neutron
This bug was fixed in the package neutron -
2:18.1.0+git2021072117.147830620f-0ubuntu2~cloud0
---
neutron (2:18.1.0+git2021072117.147830620f-0ubuntu2~cloud0) focal-xena;
urgency=medium
.
* New update for the Ubuntu Cloud Archive.
.
neutron
Thanks all for really digging into the issue!
Quite honestly reverting that one commit might have fixed the observed issue.
But having an potential ~3 second delay in the code path should not have this
impact at all.
What I am trying to say is that there might be a whole other issue with timing
I've added upstream oslo.privsep to this bug. It seems that minimally an
except block with a log message would be useful in the send_recv()
method from oslo_privsep/comm.py.
** Also affects: oslo.privsep
Importance: Undecided
Status: New
--
You received this bug notification because
This bug was fixed in the package neutron -
2:18.1.0+git2021072117.147830620f-0ubuntu2
---
neutron (2:18.1.0+git2021072117.147830620f-0ubuntu2) impish; urgency=medium
* d/p/revert-l3-ha-retry-when-setting-ha-router-gw-status.patch: Revert
upstream patch that introduced
** Changed in: cloud-archive
Status: Triaged => Fix Committed
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1927868
Title:
vRouter not working after update to 16.3.1
To manage notifications
Hello Alexandros, or anyone else affected,
Accepted neutron into hirsute-proposed. The package will build now and
be available at
https://launchpad.net/ubuntu/+source/neutron/2:18.1.0-0ubuntu2 in a few
hours, and then in the -proposed repository.
Please help us by testing this new package. See
** Description changed:
We run a juju managed Openstack Ussuri on Bionic. After updating neutron
packages from 16.3.0 to 16.3.1 all virtual routers stopped working. It
seems that most (not all) namespaces are created but have only the lo
interface and sometime the ha-XYZ interface in DOWN
** Also affects: neutron (Ubuntu Hirsute)
Importance: Undecided
Status: New
** Also affects: neutron (Ubuntu Impish)
Importance: Critical
Status: Triaged
** Also affects: neutron (Ubuntu Focal)
Importance: Undecided
Status: New
** Changed in: neutron (Ubuntu Focal)
We believe we've narrowed this down to a regression in the commit
"[L3][HA] Retry when setting HA router GW status". Reverting that patch
appears to have fixed this issue in our test environment so we are going
to move forward with an SRU for further testing. Full debug details can
be found in the
Edward Hope-Morley has recreated this and put a lot of effort into
debugging. We can triage this officially since we have a recreate that
we're debugging. We'll report back with more details soon.
** Changed in: neutron (Ubuntu)
Status: Confirmed => Triaged
** Changed in: neutron (Ubuntu)
** Changed in: neutron (Ubuntu)
Status: Incomplete => Confirmed
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1927868
Title:
vRouter not working after update to 16.3.1
To manage
On the assumption this is somewhere in the L3/HA codetree
$ git log --pretty=oneline --no-merges 15.3.2..15.3.3 neutron/agent/l3
12c07ba3ea9c6501dd7494561e2920496407c48b [L3][HA] Retry when setting HA router
GW status.
3b2b7f4fe7bacb99028b5cba7ac7a8e6c412d965 Remove update_initial_state() method
I tried L3 agent with neutron==16.3.1, problem was reproduced.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1927868
Title:
vRouter not working after update to 16.3.1
To manage notifications about
Latest build ussuri kolla neutron-l3-agent container is based on
neutron==16.3.3.dev45.
L3 agent is stuck in "Staring router update" for part of routers.
2021-06-30 14:07:50.863 575 INFO neutron.agent.l3.agent [-] Starting
router update for a9cb59da-3058-44ac-a2f7-21add6160e6d, action 3,
I've same problem on setup. We use kolla-ansible for upgrade tasks, and
after upgrade from train to ussuri problem was reproduced. Workaround is
downgrade neutron==16.3.2 to neutron==16.0.0.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to
I've marked this as incomplete for now since we haven't been able to
recreate. We'll need more information on steps that need to be taken to
recreate and triage. If anyone has more details on how to recreate
please add them to the bug and feel free to move back to New state.
--
You received this
** Changed in: neutron (Ubuntu)
Status: New => Incomplete
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1927868
Title:
vRouter not working after update to 16.3.1
To manage notifications
I have just re-tested all of this as follows:
* deployed Openstack Train (on Bionic i.e. 2:15.3.3-0ubuntu1~cloud0) with 3
gateway nodes
* created one HA router, one vm with one fip
* can ping fip and confirm single active router
* upgraded neutron-server (api) to 16.3.0-0ubuntu3~cloud0
Hello,
I reviewed the code path and upgrade in my reproducer, following the approach
of upgrading neutron-gateway and subsequently neutron-api doesn't works because
of a mismatch
in the migrations/rpc versions that causes the ha port to fail to be
created/updated,
then the keepalived process
I've had a go at deploying Train and upgrading Neutron to latest Ussuri
and I see the same issue. Looking closer what I see is that post-upgrade
Neutron l3-agent has not spawned any keepalived processes hence why no
router goes active. When the agent is restarted it would normally
receive two
Just a note that I've been running a loop for the last few hours similar
to Christian's create/delete scripts in the description of
https://bugs.launchpad.net/neutron/+bug/1916024 with no success in
recreating. This was on ussuri with neutron packages at
2:16.3.2-0ubuntu3~cloud0.
--
You received
** Changed in: neutron (Ubuntu)
Status: Triaged => New
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1927868
Title:
vRouter not working after update to 16.3.1
To manage notifications about
I haven't been able to recreate this when upgrading packages in the
correct order and running db migrations as mentioned in
https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1927868/comments/23.
Today I attempted to recreate this a few times, and my HA routers look
correct after upgrade.
@Rodolfo, We still aren't 100% certain that this particular change is
the culprit but it's highly suspect because of the nature of the change
compared to other commits.
In this ticket you have at least 4 instances of this bug being
reproducible by different people including Ubuntu maintainers and
Hello Jared:
How this change is breaking any deployment? If the interface is there,
there will be no wait at all. If the interface is not present, then you
have a problem and this is not related to the timeout.
First you need to find why, when "set_link_status" is called, the GW
interface is not
@Christian, sorry if it wasn't clear but I don't believe this is
database migration related.
@Rodolfo, At the very least the code that was written with timeout=3 is
likely problematic and should have been at least made user configurable.
It's quite possible this change is the culprit and breaks
Hello Christian:
Please check the bug and the commit description [1]. As commented in
c#4:
What the patch is solving is that situation when the interface
disappears and reappears again, while keepalived is configuring it.
If you still see this error message this is because:
- keepalived didn't
Corey, Jared I believe your analysis is running a little in the wrong
direction here:
1) We run OpenStack TRAIN (15) and also experienced the described
issues. So there cannot be any relation to the database schema upgrades.
2) We did experience the issue even before the recent upgrade and we
@Corey In our case there was no release upgrade. We have been to Ussuri
for many weeks and running happily. The problem came up only after we
update the relevant ubuntu packages from 16.3.0 to 16.3.1. Just
reverting back to 16.3.0 resolved the issue.
--
You received this bug notification because
Our order of operations went as such:
- All packages get updated to latest available Ussuri (16.3.1) for Bionic via
apt-get dist-upgrade
- Stop services (systemctl stop neutron-dhcp-agent; systemctl stop
neutron-metadata-agent; systemctl stop neutron-ovs-cleanup; systemctl stop
It seems this may only present itself if neutron-server, etc are not
upgraded and database migrations (ie. neutron-db-manage) are not run
prior to network nodes being upgraded.
In a juju deployment this means that neutron-api units need to be
upgraded prior to neutron-gateway units.
I'll test
Update on testing results from my end, based on
https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1927868/comments/19.
Attempts to upgrade from train to any of the ussuri package versions in
the ussuri-staging PPA [1] hit the bug or a similar bad state of routers
(both routers are DOWN, or
Yes, Billy Olsen, we are a little in doubt about this
(https://bugs.launchpad.net/neutron/+bug/1927868/comments/18) as well.
We have been observing such non functioning gateways on our Train
installation occasionally also before this patch / update.
Usually a "clear gateway" and a recreation via
Ante: Thank you for the link to the 16.3.0 packages. We managed to find
16.2.0 before your post and are currently using them and it seems to be
working fine for us. Should the opportunity arise we will install 16.3.0
since we've downloaded the deb files just in case.
Billy: I guess the hunt
I was able to recreate this with an active/passive scenario. The upgrade
from stein->train went well and one qg-a88fe206-a2@if19 was UP and the
other was DOWN, but after the upgrade from train->ussuri both qg-
a88fe206-a2@if19 interfaces were in DOWN state. See details attached.
** Attachment
FWIW, I don't believe that the commit referenced in comment #9 is
related to the problem, unless there's a race condition introduced by
the added delay. Looking at that commit, the code will try harder to
make sure that the device in the namespace exists before proceeding on
as it did before. The
For those looking for 16.3.0 packages, they are available at
https://launchpad.net/~ubuntu-cloud-archive/+archive/ubuntu/ussuri-
staging/+build/21146273
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
Hey Ubuntu team,
Is there any way for you to provide the Neutron 16.3.0 packages for
Bionic? We are in emergency mode here and the current 16.0.0 we are
running, although working better than 16.3.1 is causing major control
plane instability.
We are in desperate need of 16.3.0 packages.
--
You
OK, I did find evidence that this issue did begin to happen as we
upgraded our Train packages just before we upgraded to Ussuri (I believe
its best practice to update all packages, THEN upgrade to a new version
of OpenStack)
Upgraded Neutron 15.3.0 to 15.3.3 at 16:40 EDT
2021-06-10 16:40:40
keepalived: 1:1.3.9-1ubuntu0.18.04.2
from: http://archive.ubuntu.com/ubuntu bionic-updates/main
neutron-l3-agent.log filled with the following log lines for every single
router in our cluster:
2021-06-10 19:59:53.602 328191 ERROR neutron.agent.l3.ha_router [-] Gateway
interface for router
We run OpenStack Train on Ubuntu Bionic and observe similar issues with
L3-HA routers after having updated from 15.3.2 -> 15.3.3.
Currently we are still collecting evidence, but, in case there is an
issue with a certain router, we already observed that the gateway
interfaces on all nodes running
This is a bit strange as we do run Ussuri scenario jobs with L3-HA, see e.g.
https://zuul.opendev.org/t/openstack/build/508a4cc49b0844978d107b74ed18982d/logs
and it works fine. It also runs on Ubuntu Bionic.
Can You check what keepalived version do You have, maybe compare other Your
packages
@LIU, they're using the Ubuntu package which imports upstream releases
from the published tarballs at
https://tarballs.opendev.org/openstack/neutron/. So I think looking
through the delta of commits that you've listed is the right way to go
to narrow down on the offending patch.
--
You received
If we can narrow in on the the offending commit we can look at getting
an SRU out asap with the change reverted.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1927868
Title:
vRouter not working
590551d [L3][HA] Retry when setting HA router GW status.
I'm not a developer but this change does reflect the log error messages
myself and the OP are seeing.
https://github.com/openstack/neutron/commit/8f5a801270f81bd9fe3559fee9c1714c97849b3e
--
You received this bug notification because you
Seems to be a terrible bug... I have a question about juju managed
Openstack, it will directly use the tag from upstream neutron without
any private pathes or backports? If so, the following is the patches
from 13.6.0 ahead to 13.6.1, maybe we can found the problem from it.
7771f16 [L3] Delete
I was not able to roll back to 16.3.0 because they are not on any
repositories that I could find so I had to manually download neutron
debian files for focal and thankfully they seem to kind of work on
Bionic Ussuri. We are now running 16.0.0~b3~git2020041516.5f42488a9a-
0ubuntu2 which is not
I don't think so. We don't even use DVR. Our observations were basically
identical to OP's which was a telltale log entry 'Gateway interface for
router 02945b59-639b-41be-8237-3b7933b4e32d was not set up; router will
not work properly' and also that any Layer-3 agents for a router all
remained in
Could this be related to
https://bugs.launchpad.net/neutron/+bug/1928466?
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1927868
Title:
vRouter not working after update to 16.3.1
To manage
Hi, I've added upstream neutron to this bug to see if anyone is familiar
with it.
** Also affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
This bug appears to be present in 16.3.2 in the proposed repository as
well.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1927868
Title:
vRouter not working after update to 16.3.1
To manage
This is a terrible bug. 16.3.1 needs to be pulled from the repositories
immediately. This will cause all Ussuri OpenStack deployments on Bionic
to be completely down.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
Status changed to 'Confirmed' because the bug affects multiple users.
** Changed in: neutron (Ubuntu)
Status: New => Confirmed
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1927868
Title:
77 matches
Mail list logo
