[Bug 1914437] Re: [SRU] MessageTimeout and DuplicateMessage errors after udpate
More details on dates that regression landed: stein-updates - 9.5.0-0ubuntu1~cloud1 landed 2021-01-14 rocky-updates - not affected (update never made it past rocky-proposed) queens-updates - not affected (update never made it past queens-proposed) bionic-updates - 5.35.0-0ubuntu2 landed 2021-01-28 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1914437 Title: [SRU] MessageTimeout and DuplicateMessage errors after udpate To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1914437/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1914437] Re: [SRU] MessageTimeout and DuplicateMessage errors after udpate
** Changed in: oslo.messaging Status: New => Invalid -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1914437 Title: [SRU] MessageTimeout and DuplicateMessage errors after udpate To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1914437/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1914437] Re: [SRU] MessageTimeout and DuplicateMessage errors after udpate
This bug was fixed in the package python-oslo.messaging - 5.35.0-0ubuntu3~cloud0 --- python-oslo.messaging (5.35.0-0ubuntu3~cloud0) xenial-queens; urgency=medium . * New update for the Ubuntu Cloud Archive. . python-oslo.messaging (5.35.0-0ubuntu3) bionic; urgency=medium . * d/p/0001-Use-default-exchange-for-direct-messaging.patch, d/p/0002-Use-default-exchange-for-direct-messaging.patch, d/p/0003-Cancel-consumer-if-queue-down.patch: Removed as it breaks partial upgrades (LP: #1914437). ** Changed in: cloud-archive/queens Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1914437 Title: [SRU] MessageTimeout and DuplicateMessage errors after udpate To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1914437/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1914437] Re: [SRU] MessageTimeout and DuplicateMessage errors after udpate
This bug was fixed in the package python-oslo.messaging - 5.35.0-0ubuntu3 --- python-oslo.messaging (5.35.0-0ubuntu3) bionic; urgency=medium * d/p/0001-Use-default-exchange-for-direct-messaging.patch, d/p/0002-Use-default-exchange-for-direct-messaging.patch, d/p/0003-Cancel-consumer-if-queue-down.patch: Removed as it breaks partial upgrades (LP: #1914437). -- Chris MacNaughton Wed, 03 Feb 2021 14:33:27 + ** Changed in: python-oslo.messaging (Ubuntu Bionic) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1914437 Title: [SRU] MessageTimeout and DuplicateMessage errors after udpate To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1914437/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1914437] Re: [SRU] MessageTimeout and DuplicateMessage errors after udpate
queens-proposed tested successful - 1) partial upgrade of n-ovs at 5.35.0-0ubuntu3~cloud0 [1] and n-api/n-gateway at 5.35.0-0ubuntu1~cloud0 - instance creation successful 2) partial upgrade of n-api/n-gateway at 5.35.0-0ubuntu1~cloud0 [1] and n-ovs at 5.35.0-0ubuntu1~cloud3 - instance creation successful 3) test all neutron nodes at 5.35.0-0ubuntu3~cloud0 - instance creation successful ** Tags removed: verification-queens-needed ** Tags added: verification-queens-done -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1914437 Title: [SRU] MessageTimeout and DuplicateMessage errors after udpate To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1914437/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1914437] Re: [SRU] MessageTimeout and DuplicateMessage errors after udpate
This bug was fixed in the package python-oslo.messaging - 8.1.0-0ubuntu1~cloud2.1 --- python-oslo.messaging (8.1.0-0ubuntu1~cloud2.1) bionic-rocky; urgency=medium . * d/p/0001-Use-default-exchange-for-direct-messaging.patch, d/p/0002-Use-default-exchange-for-direct-messaging.patch, d/p/0003-Cancel-consumer-if-queue-down.patch, Removed as it breaks partial upgrades (LP: #1914437). ** Changed in: cloud-archive/rocky Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1914437 Title: [SRU] MessageTimeout and DuplicateMessage errors after udpate To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1914437/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1914437] Re: [SRU] MessageTimeout and DuplicateMessage errors after udpate
This bug was fixed in the package python-oslo.messaging - 9.5.0-0ubuntu1~cloud2 --- python-oslo.messaging (9.5.0-0ubuntu1~cloud2) bionic-stein; urgency=medium . * d/p/0001-Use-default-exchange-for-direct-messaging.patch, d/p/0002-Cancel-consumer-if-queue-down.patch: Removed as it breaks partial upgrades (LP: #1914437). ** Changed in: cloud-archive/stein Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1914437 Title: [SRU] MessageTimeout and DuplicateMessage errors after udpate To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1914437/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1914437] Re: [SRU] MessageTimeout and DuplicateMessage errors after udpate
verified on stein-proposed, rocky-proposed and bionic-proposed stein-proposed -- 1) partial upgrade of n-ovs at 9.5.0-0ubuntu1~cloud2 [1] and n-api/n-gateway at 9.5.0-0ubuntu1~cloud1 - instance creation successful 2) partial upgrade of n-api/n-gateway at 9.5.0-0ubuntu1~cloud2 [1] and n-ovs at 9.5.0-0ubuntu1~cloud1 - instance creation successful 3) test all neutron nodes at 9.5.0-0ubuntu1~cloud2 - instance creation successful rocky-proposed -- 1) partial upgrade of n-ovs at 8.1.0-0ubuntu1~cloud2.1 [1] and n-api/n-gateway at 8.1.0-0ubuntu1~cloud0 - instance creation successful 2) partial upgrade of n-api/n-gateway at 8.1.0-0ubuntu1~cloud0 [1] and n-ovs at 8.1.0-0ubuntu1~cloud2.1 - instance creation successful 3) test all neutron nodes at 8.1.0-0ubuntu1~cloud2.1 - instance creation successful bionic-proposed --- 1) partial upgrade of n-ovs at 5.35.0-0ubuntu3 [1] and n-api/n-gateway at 5.35.0-0ubuntu1 - instance creation successful 2) partial upgrade of n-api/n-gateway at 5.35.0-0ubuntu3 [1] and n-ovs at 5.35.0-0ubuntu1 - instance creation successful 3) partial upgrade of n-ovs at 5.35.0-0ubuntu2 [1] and n-api/n-gateway at 5.35.0-0ubuntu3 - instance creation failed as expected (see regression potential) 4) partial upgrade of n-api/n-gateway at 5.35.0-0ubuntu3 [1] and n-ovs at 5.35.0-0ubuntu2 - instance creation failed as expected (see regression potential) 5) test all neutron nodes at 5.35.0-0ubunt3 - instance creation successful ** Tags removed: verification-needed-bionic verification-queens-needed verification-rocky-needed verification-stein-needed ** Tags added: verification-done-bionic verification-rocky-done verification-stein-done ** Tags removed: verification-needed ** Tags added: verification-done ** Tags added: verification-queens-needed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1914437 Title: [SRU] MessageTimeout and DuplicateMessage errors after udpate To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1914437/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1914437] Re: [SRU] MessageTimeout and DuplicateMessage errors after udpate
** Description changed:
- A recent update to oslo.messaging to resolve bug 1789177 causes
- failures.
+ [Impact]
+ A recent update to oslo.messaging to resolve #1789177 causes failures.
(Below comments copied form the original bug):
After a partial upgrade (only one side, producers or consumers), there
are a lot of MessageTimeout and DuplicateMessage errors in the logs.
Downgrading back to 5.35.0-0ubuntu1~cloud0 fixed the problem.
Right after restarted n-ovs-agent, I can see a lot of errors in rabbitmq
log[1]
which is the same as the error when rabbitmq failover issue ( the original
issue of this LP )
Then after I upgraded oslo.messaging in neutron-api unit and restarted
neutron-server, below errors are gone and I was able to create instance
again.
After upgrading oslo.messaging in n-ovs only, exchange they communicate
didn't match.
As changing exchanges they use depends on publisher-cosumer relation.
So I think there are two ways.
1. revert this patch for Q ( original failover problem will be there )
2. upgrade them with maintenance window
Thanks a lot
[1]
=ERROR REPORT 3-Feb-2021::03:25:26 ===
Channel error on connection <0.2379.1> (10.0.0.32:60430 -> 10.0.0.34:5672,
vhost: 'openstack', user: 'neutron'), channel 1:
{amqp_error,not_found,
"no exchange 'reply_7da3cecc31b34bdeb96c866dc84e3044' in vhost
'openstack'",
'basic.publish'}
10.0.0.32 is neutron-api unit
+
+ [Test Case]
+ This SRU needs the following scenarios tested:
+
+ 1) partial upgrade of n-ovs at 5.35.0-0ubuntu3 [1] and n-api/n-gateway
+ at 5.35.0-0ubuntu1 - instance creation will be successful
+
+ 2) partial upgrade of n-api/n-gateway at 5.35.0-0ubuntu3 [1] and n-ovs
+ at 5.35.0-0ubuntu1 - instance creation will be successful
+
+ 3) partial upgrade of n-ovs at 5.35.0-0ubuntu2 [1] and n-api/n-gateway
+ at 5.35.0-0ubuntu3 - instance creation will fail (see regression
+ potential)
+
+ 4) partial upgrade of n-api/n-gateway at 5.35.0-0ubuntu3 [1] and n-ovs
+ at 5.35.0-0ubuntu2 - instance creation will fail (see regression
+ potential)
+
+ 5) test all neutron nodes at 5.35.0-0ubunt3 - instance creation will be
+ successful
+
+ [1] and neutron* services restarted
+
+ [Regression Potential]
+ There is regression potential for clouds that have already upgraded to
5.35.0-0ubuntu2. This needs to be tested but if a cloud has fully upgraded to
5.35.0-0ubuntu2, then the same disruption that this SRU is trying to solve may
once again occur in a cloud with some services running 5.35.0-0ubuntu2 and some
running 5.35.0-0ubuntu3. Once that cloud is entirely at 5.35.0-0ubuntu3,
messages will no longer timeout.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1914437
Title:
[SRU] MessageTimeout and DuplicateMessage errors after udpate
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1914437/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1914437] Re: [SRU] MessageTimeout and DuplicateMessage errors after udpate
Hello Chris, or anyone else affected, Accepted python-oslo.messaging into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source /python-oslo.messaging/5.35.0-0ubuntu3 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed- bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification- failed-bionic. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Changed in: python-oslo.messaging (Ubuntu Bionic) Status: Triaged => Fix Committed ** Tags added: verification-needed verification-needed-bionic -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1914437 Title: [SRU] MessageTimeout and DuplicateMessage errors after udpate To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1914437/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1914437] Re: [SRU] MessageTimeout and DuplicateMessage errors after udpate
Sorry, I specified the tag name to Chris wrong.
** Tags removed: regression-updates
** Tags added: regression-update
** Description changed:
- A recent update to oslo.messaging to resolve #1789177 causes failures.
+ A recent update to oslo.messaging to resolve bug 1789177 causes
+ failures.
(Below comments copied form the original bug):
After a partial upgrade (only one side, producers or consumers), there
are a lot of MessageTimeout and DuplicateMessage errors in the logs.
Downgrading back to 5.35.0-0ubuntu1~cloud0 fixed the problem.
Right after restarted n-ovs-agent, I can see a lot of errors in rabbitmq
log[1]
which is the same as the error when rabbitmq failover issue ( the original
issue of this LP )
Then after I upgraded oslo.messaging in neutron-api unit and restarted
neutron-server, below errors are gone and I was able to create instance
again.
After upgrading oslo.messaging in n-ovs only, exchange they communicate
didn't match.
As changing exchanges they use depends on publisher-cosumer relation.
So I think there are two ways.
1. revert this patch for Q ( original failover problem will be there )
2. upgrade them with maintenance window
Thanks a lot
[1]
=ERROR REPORT 3-Feb-2021::03:25:26 ===
Channel error on connection <0.2379.1> (10.0.0.32:60430 -> 10.0.0.34:5672,
vhost: 'openstack', user: 'neutron'), channel 1:
{amqp_error,not_found,
- "no exchange 'reply_7da3cecc31b34bdeb96c866dc84e3044' in vhost
'openstack'",
- 'basic.publish'}
+ "no exchange 'reply_7da3cecc31b34bdeb96c866dc84e3044' in vhost
'openstack'",
+ 'basic.publish'}
10.0.0.32 is neutron-api unit
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1914437
Title:
[SRU] MessageTimeout and DuplicateMessage errors after udpate
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1914437/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1914437] Re: [SRU] MessageTimeout and DuplicateMessage errors after udpate
** Also affects: cloud-archive/queens Importance: Undecided Status: New ** Changed in: cloud-archive/queens Status: New => Incomplete ** Changed in: cloud-archive/queens Status: Incomplete => Triaged ** Changed in: cloud-archive/queens Importance: Undecided => Critical -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1914437 Title: [SRU] MessageTimeout and DuplicateMessage errors after udpate To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1914437/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1914437] Re: [SRU] MessageTimeout and DuplicateMessage errors after udpate
I'm marking this as not affecting Queens because the change that caused this regression didn't get out of queens-proposed. ** No longer affects: cloud-archive/queens -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1914437 Title: [SRU] MessageTimeout and DuplicateMessage errors after udpate To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1914437/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1914437] Re: [SRU] MessageTimeout and DuplicateMessage errors after udpate
** Also affects: python-oslo.messaging (Ubuntu) Importance: Undecided Status: New ** Also affects: python-oslo.messaging (Ubuntu Bionic) Importance: Undecided Status: New ** Changed in: python-oslo.messaging (Ubuntu Bionic) Status: New => Triaged ** Changed in: python-oslo.messaging (Ubuntu Bionic) Importance: Undecided => Critical ** Changed in: python-oslo.messaging (Ubuntu) Status: New => Invalid ** Also affects: cloud-archive Importance: Undecided Status: New ** Also affects: cloud-archive/rocky Importance: Undecided Status: New ** Also affects: cloud-archive/stein Importance: Undecided Status: New ** Also affects: cloud-archive/queens Importance: Undecided Status: New ** Changed in: cloud-archive/stein Status: New => Triaged ** Changed in: cloud-archive/rocky Status: New => Triaged ** Changed in: cloud-archive/queens Status: New => Triaged ** Changed in: cloud-archive/stein Importance: Undecided => Critical ** Changed in: cloud-archive/rocky Importance: Undecided => Critical ** Changed in: cloud-archive/queens Importance: Undecided => Critical ** Changed in: cloud-archive Status: New => Invalid -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1914437 Title: [SRU] MessageTimeout and DuplicateMessage errors after udpate To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1914437/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
