[Bug 1907686] Re: ovn: instance unable to retrieve metadata
This bug was fixed in the package openvswitch - 2.13.3-0ubuntu0.20.04.1~cloud0 --- openvswitch (2.13.3-0ubuntu0.20.04.1~cloud0) bionic-ussuri; urgency=medium . * New update for the Ubuntu Cloud Archive. . openvswitch (2.13.3-0ubuntu0.20.04.1) focal; urgency=medium . [ James Page ] * New upstream point release (LP: #1920141, LP: #1907686). * Dropped security patches, included in release: - CVE-2015-8011.patch - CVE-2020-27827.patch - CVE-2020-35498.patch * Add BD on libdbus-1-dev to resolve linking issues for DPDK builds due to changes in DPDK. * d/control: Set minimum version of libdpdk-dev to avoid build failures with 19.11.6-0ubuntu0.20.04.1. . [ Frode Nordahl ] * Fix recording of FQDN/hostname on startup (LP: #1915829): - d/p/ovs-dev-ovs-ctl-Allow-recording-hostname-separately.patch: Cherry pick of committed upstream fix to support skip of hostname configuration on ovs-vswitchd/ovsdb-server startup. - d/openvswitch-switch.ovs-record-hostname.service: Record hostname in Open vSwitch after network-online.target using new systemd unit. - d/openvswitch-switch.ovs-vswitchd.service: Pass `--no-record-hostname` option to `ovs-ctl` to delegate recording of hostname to the separate service. - d/openvswitch-switch.ovsdb-server.service: Pass `--no-record-hostname` option to `ovs-ctl` to delegate recording of hostname to the separate service. - d/openvswitch-switch.service: Add `Also` reference to ovs-record-hostname.service so that the service is enabled on install. - d/rules: Add `ovs-record-hostname.service` to package build. ** Changed in: cloud-archive/ussuri Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907686 Title: ovn: instance unable to retrieve metadata To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907686] Re: ovn: instance unable to retrieve metadata
This is awaiting verification of https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1915829 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907686 Title: ovn: instance unable to retrieve metadata To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907686] Re: ovn: instance unable to retrieve metadata
** Changed in: cloud-archive/victoria Status: New => Won't Fix -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907686 Title: ovn: instance unable to retrieve metadata To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907686] Re: ovn: instance unable to retrieve metadata
This bug was fixed in the package openvswitch - 2.13.3-0ubuntu0.20.04.1 --- openvswitch (2.13.3-0ubuntu0.20.04.1) focal; urgency=medium [ James Page ] * New upstream point release (LP: #1920141, LP: #1907686). * Dropped security patches, included in release: - CVE-2015-8011.patch - CVE-2020-27827.patch - CVE-2020-35498.patch * Add BD on libdbus-1-dev to resolve linking issues for DPDK builds due to changes in DPDK. * d/control: Set minimum version of libdpdk-dev to avoid build failures with 19.11.6-0ubuntu0.20.04.1. [ Frode Nordahl ] * Fix recording of FQDN/hostname on startup (LP: #1915829): - d/p/ovs-dev-ovs-ctl-Allow-recording-hostname-separately.patch: Cherry pick of committed upstream fix to support skip of hostname configuration on ovs-vswitchd/ovsdb-server startup. - d/openvswitch-switch.ovs-record-hostname.service: Record hostname in Open vSwitch after network-online.target using new systemd unit. - d/openvswitch-switch.ovs-vswitchd.service: Pass `--no-record-hostname` option to `ovs-ctl` to delegate recording of hostname to the separate service. - d/openvswitch-switch.ovsdb-server.service: Pass `--no-record-hostname` option to `ovs-ctl` to delegate recording of hostname to the separate service. - d/openvswitch-switch.service: Add `Also` reference to ovs-record-hostname.service so that the service is enabled on install. - d/rules: Add `ovs-record-hostname.service` to package build. -- James Page Wed, 24 Mar 2021 08:46:41 + ** Changed in: openvswitch (Ubuntu Focal) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907686 Title: ovn: instance unable to retrieve metadata To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907686] Re: ovn: instance unable to retrieve metadata
This bug was fixed in the package openvswitch - 2.13.3-0ubuntu0.20.10.1 --- openvswitch (2.13.3-0ubuntu0.20.10.1) groovy; urgency=medium [ James Page ] * New upstream point release (LP: #1920141, LP: #1907686): - d/p/*: Refresh including drop of part of patch as equivalent changes included in point release. - Dropped security patches, included in release: + CVE-2015-8011.patch + CVE-2020-27827.patch + CVE-2020-35498.patch * Add BD on libdbus-1-dev to resolve linking issues for DPDK builds due to changes in DPDK. * d/control: Set minimum version of libdpdk-dev to avoid build failures with 19.11.6-0ubuntu0.20.10.1. [ Frode Nordahl ] * Fix recording of FQDN/hostname on startup (LP: #1915829): - d/p/ovs-dev-ovs-ctl-Allow-recording-hostname-separately.patch: Cherry pick of committed upstream fix to support skip of hostname configuration on ovs-vswitchd/ovsdb-server startup. - d/openvswitch-switch.ovs-record-hostname.service: Record hostname in Open vSwitch after network-online.target using new systemd unit. - d/openvswitch-switch.ovs-vswitchd.service: Pass `--no-record-hostname` option to `ovs-ctl` to delegate recording of hostname to the separate service. - d/openvswitch-switch.ovsdb-server.service: Pass `--no-record-hostname` option to `ovs-ctl` to delegate recording of hostname to the separate service. - d/openvswitch-switch.service: Add `Also` reference to ovs-record-hostname.service so that the service is enabled on install. - d/rules: Add `ovs-record-hostname.service` to package build. -- James Page Wed, 24 Mar 2021 08:46:06 + ** Changed in: openvswitch (Ubuntu Groovy) Status: Fix Committed => Fix Released ** CVE added: https://cve.mitre.org/cgi-bin/cvename.cgi?name=2015-8011 ** CVE added: https://cve.mitre.org/cgi-bin/cvename.cgi?name=2020-27827 ** CVE added: https://cve.mitre.org/cgi-bin/cvename.cgi?name=2020-35498 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907686 Title: ovn: instance unable to retrieve metadata To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907686] Re: ovn: instance unable to retrieve metadata
** Tags removed: verification-needed ** Tags added: verification-done -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907686 Title: ovn: instance unable to retrieve metadata To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907686] Re: ovn: instance unable to retrieve metadata
Deployed groovy from proposed; restarted all ovn-central daemons post deployment and then started a new instance. Instance booted and correctly retrieved metadata: 2021-05-13 12:46:06.269 2619 INFO eventlet.wsgi.server [-] 192.168.0.83, "GET /openstack/2018-08-27/network_data.json HTTP/1.1" status: 200 len: 490 time: 0.0040092 ** Tags removed: verification-needed-groovy ** Tags added: verification-done-groovy -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907686 Title: ovn: instance unable to retrieve metadata To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907686] Re: ovn: instance unable to retrieve metadata
Yep on that today - thanks for the poke @sil2100 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907686 Title: ovn: instance unable to retrieve metadata To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907686] Re: ovn: instance unable to retrieve metadata
Hello! Could we get this verified for groovy? Thanks! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907686 Title: ovn: instance unable to retrieve metadata To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907686] Re: ovn: instance unable to retrieve metadata
Another customer hit this bug - waiting for release on uca ussuri -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907686 Title: ovn: instance unable to retrieve metadata To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907686] Re: ovn: instance unable to retrieve metadata
Typo in #26, tested in ussuri-proposed (not bionic-proposed) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907686 Title: ovn: instance unable to retrieve metadata To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907686] Re: ovn: instance unable to retrieve metadata
After applying the fix in bionic-proposed,
only one compute node exhibited the logs mentioned in #5
2021-01-12 06:48:49.848 52569 ERROR neutron.agent.ovn.metadata.server [-]
Unexpected error.: AttributeError: 'MetadataProxyHandler' object has no
attribute 'sb_idl'
sb_idl is never initialised as the connection timed out during initialisation
phase.
See error log:
2021-04-15 22:30:08.542 69188 ERROR
neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] OVS database
connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS
and OVN services are available and that the 'ovn_nb_connection' and
'ovn_sb_connection' configuration options are correct.: Exception: Timeout
2021-04-15 22:30:08.542 69188 ERROR
neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn Traceback (most
recent call last):
2021-04-15 22:30:08.542 69188 ERROR
neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File
"/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py",
line 67, in start_connection
2021-04-15 22:30:08.542 69188 ERROR
neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn
self.ovsdb_connection.start()
2021-04-15 22:30:08.542 69188 ERROR
neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File
"/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line
79, in start
2021-04-15 22:30:08.542 69188 ERROR
neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn
idlutils.wait_for_change(self.idl, self.timeout)
2021-04-15 22:30:08.542 69188 ERROR
neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File
"/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line
173, in wait_for_change
2021-04-15 22:30:08.542 69188 ERROR
neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn raise
Exception("Timeout") # TODO(twilson) use TimeoutException?
2021-04-15 22:30:08.542 69188 ERROR
neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn Exception:
Timeout
2021-04-15 22:30:08.542 69188 ERROR
neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager [-] Error
during notification for
neutron.agent.ovn.metadata.server.MetadataProxyHandler.post_fork_initialize-476074
process, after_init:
neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable:
OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify
that the OVS and OVN services are available and that the 'ovn_nb_connection'
and 'ovn_sb_connection' configuration options are correct.
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Traceback
(most recent call last):
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File
"/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py",
line 67, in start_connection
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager
self.ovsdb_connection.start()
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File
"/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line
79, in start
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager
idlutils.wait_for_change(self.idl, self.timeout)
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File
"/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line
173, in wait_for_change
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager raise
Exception("Timeout") # TODO(twilson) use TimeoutException?
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Exception:
Timeout
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1907686
Title:
ovn: instance unable to retrieve metadata
To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907686] Re: ovn: instance unable to retrieve metadata
Regression testing results for this were successful (apologies for the delay): == focal-ussuri == 21:38:43 == 21:38:43 Totals 21:38:43 == 21:38:43 Ran: 97 tests in 1663.4194 sec. 21:38:43 - Passed: 89 21:38:43 - Skipped: 8 21:38:43 - Expected Fail: 0 21:38:43 - Unexpected Success: 0 21:38:43 - Failed: 0 21:38:43 Sum of execute time for each test: 1151.4881 sec. == bionic-ussuri == 02:32:56 == 02:32:56 Totals 02:32:56 == 02:32:56 Ran: 97 tests in 960.0834 sec. 02:32:56 - Passed: 89 02:32:56 - Skipped: 8 02:32:56 - Expected Fail: 0 02:32:56 - Unexpected Success: 0 02:32:56 - Failed: 0 02:32:56 Sum of execute time for each test: 827.3194 sec. == groovy-victoria == 00:58:25 == 00:58:25 Totals 00:58:25 == 00:58:25 Ran: 97 tests in 1343.5539 sec. 00:58:25 - Passed: 89 00:58:25 - Skipped: 8 00:58:25 - Expected Fail: 0 00:58:25 - Unexpected Success: 0 00:58:25 - Failed: 0 00:58:25 Sum of execute time for each test: 1143.6704 sec. == focal-victoria == 04:39:56 == 04:39:56 Totals 04:39:56 == 04:39:56 Ran: 97 tests in 1445.2159 sec. 04:39:56 - Passed: 89 04:39:56 - Skipped: 8 04:39:56 - Expected Fail: 0 04:39:56 - Unexpected Success: 0 04:39:56 - Failed: 0 04:39:56 Sum of execute time for each test: 1045.0895 sec. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907686 Title: ovn: instance unable to retrieve metadata To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907686] Re: ovn: instance unable to retrieve metadata
Verified bionic-ussuri-proposed python3-openvswitch package in production environment resolved the issue. ** Tags removed: verification-ussuri-needed ** Tags added: verification-ussuri-done -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907686 Title: ovn: instance unable to retrieve metadata To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907686] Re: ovn: instance unable to retrieve metadata
Verified focal-ussuri in a converged networking environment. ** Tags removed: verification-needed-focal ** Tags added: verification-done-focal -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907686 Title: ovn: instance unable to retrieve metadata To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907686] Re: ovn: instance unable to retrieve metadata
@Robie, I have been unable to validate as the amd64 package is failing to build: https://launchpad.net/ubuntu/+source/openvswitch/2.13.3-0ubuntu0.20.04.1/+build/21397125 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907686 Title: ovn: instance unable to retrieve metadata To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907686] Re: ovn: instance unable to retrieve metadata
Hello James, or anyone else affected, Accepted openvswitch into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openvswitch/2.13.3-0ubuntu0.20.10.1 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed- groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification- failed-groovy. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Changed in: openvswitch (Ubuntu Groovy) Status: Triaged => Fix Committed ** Tags added: verification-needed verification-needed-groovy ** Changed in: openvswitch (Ubuntu Focal) Status: Triaged => Fix Committed ** Tags added: verification-needed-focal -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907686 Title: ovn: instance unable to retrieve metadata To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907686] Re: ovn: instance unable to retrieve metadata
For visibility: I've filed a bug for adding monitoring of the Metadata agent via the HTTP layer: LP#1921653 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907686 Title: ovn: instance unable to retrieve metadata To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907686] Re: ovn: instance unable to retrieve metadata
** Description changed: + [Impact] + Cloud instances are unable to retrieve metadata on startup. + + [Test Case] + Deploy OpenStack with OVN/OVS + Restart OVN central controllers + Create a new instance + Instance will fail to retrieve metadata with the message from the original bug report displayed in the metadata agent log on the local hypervisor + + [Regression Potential] + The fix for this issue is included in the upstream 2.13.3 release of OVS. + The fix ensures that SSL related connection issues are correctly handling in python3-ovs avoiding an issue where the connection to the OVN SB IDL is reset and never recreated. + The OVN drivers use python3-ovsdbapp which in turn bases off code provided by python3-ovs. + + + [Original Bug Report] Ubuntu:focal OpenStack: ussuri Instance port: hardware offloaded instance created, attempts to access metadata - metadata agent can't resolve the port/network combination: 2020-12-10 15:00:18.258 4732 INFO neutron.agent.ovn.metadata.agent [-] Port d65418a6-d0e9-47e6-84ba-3d02fe75131a in datapath 37706e4d-ce2a-4d81-8c61-3fd12437a0a7 bound to our ch assis 2020-12-10 15:00:31.672 8062 ERROR neutron.agent.ovn.metadata.server [-] No port found in network 37706e4d-ce2a-4d81-8c61-3fd12437a0a7 with IP address 10.5.1.155 2020-12-10 15:00:31.673 8062 INFO eventlet.wsgi.server [-] 10.5.1.155, "GET /openstack HTTP/1.1" status: 404 len: 297 time: 0.0043790 2020-12-10 15:00:34.639 8062 ERROR neutron.agent.ovn.metadata.server [-] No port found in network 37706e4d-ce2a-4d81-8c61-3fd12437a0a7 with IP address 10.5.1.155 2020-12-10 15:00:34.639 8062 INFO eventlet.wsgi.server [-] 10.5.1.155, "GET /openstack HTTP/1.1" status: 404 len: 297 time: 0.0040138 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907686 Title: ovn: instance unable to retrieve metadata To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907686] Re: ovn: instance unable to retrieve metadata
Packages have now built in: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3690 DPDK needed a fix for a regression introduced in the previous -update. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907686 Title: ovn: instance unable to retrieve metadata To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907686] Re: ovn: instance unable to retrieve metadata
The SRU for OVS 2.13.3 is tracked under bug 1920141 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907686 Title: ovn: instance unable to retrieve metadata To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907686] Re: ovn: instance unable to retrieve metadata
2.13.3 uploaded for focal and groovy to: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3690 for testing. ** No longer affects: openvswitch (Ubuntu Bionic) ** Also affects: cloud-archive Importance: Undecided Status: New ** Also affects: cloud-archive/ussuri Importance: Undecided Status: New ** Also affects: cloud-archive/victoria Importance: Undecided Status: New ** Also affects: cloud-archive/wallaby Importance: Undecided Status: New ** Changed in: cloud-archive/wallaby Status: New => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907686 Title: ovn: instance unable to retrieve metadata To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907686] Re: ovn: instance unable to retrieve metadata
** Also affects: openvswitch (Ubuntu Bionic) Importance: Undecided Status: New ** Also affects: neutron (Ubuntu Bionic) Importance: Undecided Status: New ** Also affects: openvswitch (Ubuntu Hirsute) Importance: High Status: Triaged ** Also affects: neutron (Ubuntu Hirsute) Importance: Undecided Status: Invalid ** Also affects: openvswitch (Ubuntu Groovy) Importance: Undecided Status: New ** Also affects: neutron (Ubuntu Groovy) Importance: Undecided Status: New ** Also affects: openvswitch (Ubuntu Focal) Importance: Undecided Status: New ** Also affects: neutron (Ubuntu Focal) Importance: Undecided Status: New ** No longer affects: neutron (Ubuntu) ** No longer affects: neutron (Ubuntu Bionic) ** No longer affects: neutron (Ubuntu Focal) ** No longer affects: neutron (Ubuntu Groovy) ** No longer affects: neutron (Ubuntu Hirsute) ** Changed in: openvswitch (Ubuntu Hirsute) Status: Triaged => Fix Released ** Changed in: openvswitch (Ubuntu Groovy) Status: New => Triaged ** Changed in: openvswitch (Ubuntu Focal) Status: New => Triaged ** Changed in: openvswitch (Ubuntu Bionic) Status: New => Triaged ** Changed in: openvswitch (Ubuntu Focal) Importance: Undecided => High ** Changed in: openvswitch (Ubuntu Groovy) Importance: Undecided => High ** Changed in: openvswitch (Ubuntu Bionic) Importance: Undecided => High -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907686 Title: ovn: instance unable to retrieve metadata To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907686] Re: ovn: instance unable to retrieve metadata
It was actually already fixed upstream in [0] and backported to branch-2.13 in [1] which is tagged in v2.13.3. 0: https://mail.openvswitch.org/pipermail/ovs-dev/2020-September/375050.html 1: https://github.com/openvswitch/ovs/commit/cdf2cf98e31ea4c99f62bf7a2c57a22050a71bbd So this would be fixed by a new point release update in Ubuntu. ** Changed in: openvswitch (Ubuntu) Status: New => Triaged ** Changed in: openvswitch (Ubuntu) Importance: Undecided => High ** Changed in: neutron (Ubuntu) Status: Confirmed => Invalid ** Changed in: neutron Status: New => Invalid ** Changed in: charm-ovn-chassis Status: New => Invalid -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907686 Title: ovn: instance unable to retrieve metadata To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907686] Re: ovn: instance unable to retrieve metadata
With the proposed fix in place local validation allows me to successfully create instances just after a full cloud reboot as well as after synthetic test of restarting all the OVN ovsdb-server processes at once multiple times. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907686 Title: ovn: instance unable to retrieve metadata To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907686] Re: ovn: instance unable to retrieve metadata
Fix proposed upstream: http://patchwork.ozlabs.org/project/openvswitch/patch/20210316123957.3289320-1-frode.nord...@canonical.com/ -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907686 Title: ovn: instance unable to retrieve metadata To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907686] Re: ovn: instance unable to retrieve metadata
While looking to reproduce this issue we found this in the logs of both
the neutron-ovn-metadata-agent and neutron-server:
2021-03-16 10:56:45.465 270932 ERROR ovsdbapp.backend.ovs_idl.connection [-]
(111, 'ECONNREFUSED'): OpenSSL.SSL.SysCallError: (111, 'ECONNREFUSED')
2021-03-16 10:56:45.465 270932 ERROR ovsdbapp.backend.ovs_idl.connection
Traceback (most recent call last):
2021-03-16 10:56:45.465 270932 ERROR ovsdbapp.backend.ovs_idl.connection File
"/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line
106, in run
2021-03-16 10:56:45.465 270932 ERROR ovsdbapp.backend.ovs_idl.connection
self.idl.run()
2021-03-16 10:56:45.465 270932 ERROR ovsdbapp.backend.ovs_idl.connection File
"/usr/lib/python3/dist-packages/ovs/db/idl.py", line 240, in run
2021-03-16 10:56:45.465 270932 ERROR ovsdbapp.backend.ovs_idl.connection
self._session.run()
2021-03-16 10:56:45.465 270932 ERROR ovsdbapp.backend.ovs_idl.connection File
"/usr/lib/python3/dist-packages/ovs/jsonrpc.py", line 532, in run
2021-03-16 10:56:45.465 270932 ERROR ovsdbapp.backend.ovs_idl.connection
self.__connect()
2021-03-16 10:56:45.465 270932 ERROR ovsdbapp.backend.ovs_idl.connection File
"/usr/lib/python3/dist-packages/ovs/jsonrpc.py", line 467, in __connect
2021-03-16 10:56:45.465 270932 ERROR ovsdbapp.backend.ovs_idl.connection
error, self.stream = ovs.stream.Stream.open(name)
2021-03-16 10:56:45.465 270932 ERROR ovsdbapp.backend.ovs_idl.connection File
"/usr/lib/python3/dist-packages/ovs/stream.py", line 192, in open
2021-03-16 10:56:45.465 270932 ERROR ovsdbapp.backend.ovs_idl.connection
err = ovs.socket_util.check_connection_completion(sock)
2021-03-16 10:56:45.465 270932 ERROR ovsdbapp.backend.ovs_idl.connection File
"/usr/lib/python3/dist-packages/ovs/socket_util.py", line 181, in
check_connection_completion
2021-03-16 10:56:45.465 270932 ERROR ovsdbapp.backend.ovs_idl.connection
sock.send("\0".encode(), socket.MSG_DONTWAIT)
2021-03-16 10:56:45.465 270932 ERROR ovsdbapp.backend.ovs_idl.connection File
"/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1737, in send
2021-03-16 10:56:45.465 270932 ERROR ovsdbapp.backend.ovs_idl.connection
self._raise_ssl_error(self._ssl, result)
2021-03-16 10:56:45.465 270932 ERROR ovsdbapp.backend.ovs_idl.connection File
"/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1639, in _raise_ssl_error
2021-03-16 10:56:45.465 270932 ERROR ovsdbapp.backend.ovs_idl.connection
raise SysCallError(errno, errorcode.get(errno))
2021-03-16 10:56:45.465 270932 ERROR ovsdbapp.backend.ovs_idl.connection
OpenSSL.SSL.SysCallError: (111, 'ECONNREFUSED')
2021-03-16 10:56:45.465 270932 ERROR ovsdbapp.backend.ovs_idl.connection
I suspect the log excerpts printed in comments above is fallout from
this condition and that both issues will go away with a fixed python-
ovs.
** Also affects: openvswitch (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1907686
Title:
ovn: instance unable to retrieve metadata
To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907686] Re: ovn: instance unable to retrieve metadata
We also saw this after a round of package upgrades on a cloud. python3-neutron went from 2:16.2.0-0ubuntu2~cloud0 to 2:16.2.0-0ubuntu3~cloud0 and ovn packages went from 20.03.1-0ubuntu1.1~cloud0 to 20.03.1-0ubuntu1.2~cloud0 (i.e. not an upstream version change). -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907686 Title: ovn: instance unable to retrieve metadata To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907686] Re: ovn: instance unable to retrieve metadata
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: neutron (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907686 Title: ovn: instance unable to retrieve metadata To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907686] Re: ovn: instance unable to retrieve metadata
The customer also faced this issue ** Tags added: sts -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907686 Title: ovn: instance unable to retrieve metadata To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907686] Re: ovn: instance unable to retrieve metadata
** Tags added: ovn -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907686 Title: ovn: instance unable to retrieve metadata To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907686] Re: ovn: instance unable to retrieve metadata
Also see a few of these: 2021-01-12 06:48:21.947 52569 INFO eventlet.wsgi.server [-] 10.5.3.232, "GET /openstack HTTP/1.1" status: 500 len: 362 time: 0.0006430 2021-01-12 06:48:49.848 52569 ERROR neutron.agent.ovn.metadata.server [-] Unexpected error.: AttributeError: 'MetadataProxyHandler' object has no attribute 'sb_idl' 2021-01-12 06:48:49.848 52569 ERROR neutron.agent.ovn.metadata.server Traceback (most recent call last): 2021-01-12 06:48:49.848 52569 ERROR neutron.agent.ovn.metadata.server File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/server.py", line 67, in __call__ 2021-01-12 06:48:49.848 52569 ERROR neutron.agent.ovn.metadata.server instance_id, project_id = self._get_instance_and_project_id(req) 2021-01-12 06:48:49.848 52569 ERROR neutron.agent.ovn.metadata.server File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/server.py", line 84, in _get_instance_and_project_id 2021-01-12 06:48:49.848 52569 ERROR neutron.agent.ovn.metadata.server ports = self.sb_idl.get_network_port_bindings_by_ip(network_id, 2021-01-12 06:48:49.848 52569 ERROR neutron.agent.ovn.metadata.server AttributeError: 'MetadataProxyHandler' object has no attribute 'sb_idl' 2021-01-12 06:48:49.848 52569 ERROR neutron.agent.ovn.metadata.server Restarting the neutron-ovn-metadata-agent service resolves the issue. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907686 Title: ovn: instance unable to retrieve metadata To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
