Public bug reported: Neutron Metadata agent fails to register in the Epoxy release.
On a fresh deployment, it just never registers When upgrading from Caracal, it gets marked as down because no heartbeats are sent after the upgrade. I have not tested Dalmatian. Even though heartbeats fail, it still works fine. It responds and requests can be seen in the logs. Tested in a Kayobe AIO (Kolla containers) Tested on Rocky 9 and Ubuntu Noble, both are affected OVN is unaffected Appears to have affected Kolla-Ansible since mid-march: https://review.opendev.org/c/openstack/kolla-ansible/+/944194 Severity is high. No functional issues but lots of alerts firing . Logs ==== Neutron server error (last heartbeat is time of upgrade): Agent healthcheck: found 1 dead agents out of 4: Type Last heartbeat host Metadata agent 2025-06-09 13:29:05 alex-aio-epoxy-rocky06.novalocal Openstack view after upgrade: $ openstack network agent list -c "Agent Type" -c Alive -c State -c Binary +--------------------+-------+-------+---------------------------+ | Agent Type | Alive | State | Binary | +--------------------+-------+-------+---------------------------+ | Open vSwitch agent | :-) | UP | neutron-openvswitch-agent | | DHCP agent | :-) | UP | neutron-dhcp-agent | | L3 agent | :-) | UP | neutron-l3-agent | | Metadata agent | XXX | UP | neutron-metadata-agent | +--------------------+-------+-------+---------------------------+ Neutron metadata agent response log: 2025-06-10 15:18:24.306 7 DEBUG neutron.agent.metadata.agent [None req-49747846-774c-40ad-9fef-0eaaf7ea1705 - - - - - -] Request: GET /latest/meta-data/public-ipv4/ HTTP/1.1 host: 169.254.169.254 user-agent: curl/7.84.0 accept: */* x-neutron-network-id: 03739e2a-f552-43b1-9d6f-43db61d03aaf x-forwarded-for: 10.0.0.59 connection: close handle /var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/agent/metadata/agent.py:163 2025-06-10 15:18:24.309 7 DEBUG oslo_concurrency.lockutils [None req-49747846-774c-40ad-9fef-0eaaf7ea1705 - - - - - -] Acquiring lock "oslo_read_shm_alex-aio-epoxy-rocky-7.novalocal_neutron-metadata-agent" by "oslo_messaging._drivers.amqpdriver.QManager.get.<locals>.read_from_shm" inner /var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_concurrency/lockutils.py:405 2025-06-10 15:18:24.309 7 DEBUG oslo_concurrency.lockutils [None req-49747846-774c-40ad-9fef-0eaaf7ea1705 - - - - - -] Lock "oslo_read_shm_alex-aio-epoxy-rocky-7.novalocal_neutron-metadata-agent" acquired by "oslo_messaging._drivers.amqpdriver.QManager.get.<locals>.read_from_shm" :: waited 0.001s inner /var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_concurrency/lockutils.py:410 2025-06-10 15:18:24.311 7 DEBUG oslo_concurrency.lockutils [None req-49747846-774c-40ad-9fef-0eaaf7ea1705 - - - - - -] Lock "oslo_read_shm_alex-aio-epoxy-rocky-7.novalocal_neutron-metadata-agent" "released" by "oslo_messaging._drivers.amqpdriver.QManager.get.<locals>.read_from_shm" :: held 0.001s inner /var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_concurrency/lockutils.py:424 2025-06-10 15:18:24.833 7 DEBUG neutron.agent.metadata.agent [None req-49747846-774c-40ad-9fef-0eaaf7ea1705 - - - - - -] Got ports for remote_address 10.0.0.59, network_id 03739e2a-f552-43b1-9d6f-43db61d03aaf, remote_mac None, router_id None[{'id': 'cdbb6292-e301-4f5c-bc34-33e139eb97fb', 'name': '', 'network_id': '03739e2a-f552-43b1-9d6f-43db61d03aaf', 'tenant_id': '961e2027d2ac49d8b7e76065b42206ac', 'mac_address': 'fa:16:3e:d5:a3:a6', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': 'b68cc942-cfc2-4ee7-be20-5ce5e1f62353', 'device_owner': 'compute:nova', 'standard_attr_id': 662, 'fixed_ips': [{'subnet_id': 'c0cfe0fc-f3af-4ac1-9e11-a4fbbcd31167', 'ip_address': '10.0.0.59'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': ['f2f888b6-1209-4127-8981-acec08ec4eed'], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'alex-aio-epoxy-rocky-7.novalocal', 'binding:vif_type': 'ovs', 'binding:vif_details': {'connectivity': 'l2', 'port_filter': True, 'ovs_hybrid_plug': True, 'datapath_type': 'system', 'bridge_name': 'br-int', 'bound_drivers': {'0': 'openvswitch'}}, 'port_security_enabled': True, 'tags': [], 'created_at': '2025-06-10T14:57:42Z', 'updated_at': '2025-06-10T14:57:47Z', 'revision_number': 5, 'project_id': '961e2027d2ac49d8b7e76065b42206ac'}] get_port /var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/agent/metadata/agent.py:255 ** Affects: neutron Importance: Undecided Status: New ** Tags: ovs -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2113923 Title: Neutron metadata agent fails to register in Epoxy Status in neutron: New Bug description: Neutron Metadata agent fails to register in the Epoxy release. On a fresh deployment, it just never registers When upgrading from Caracal, it gets marked as down because no heartbeats are sent after the upgrade. I have not tested Dalmatian. Even though heartbeats fail, it still works fine. It responds and requests can be seen in the logs. Tested in a Kayobe AIO (Kolla containers) Tested on Rocky 9 and Ubuntu Noble, both are affected OVN is unaffected Appears to have affected Kolla-Ansible since mid-march: https://review.opendev.org/c/openstack/kolla-ansible/+/944194 Severity is high. No functional issues but lots of alerts firing . Logs ==== Neutron server error (last heartbeat is time of upgrade): Agent healthcheck: found 1 dead agents out of 4: Type Last heartbeat host Metadata agent 2025-06-09 13:29:05 alex-aio-epoxy-rocky06.novalocal Openstack view after upgrade: $ openstack network agent list -c "Agent Type" -c Alive -c State -c Binary +--------------------+-------+-------+---------------------------+ | Agent Type | Alive | State | Binary | +--------------------+-------+-------+---------------------------+ | Open vSwitch agent | :-) | UP | neutron-openvswitch-agent | | DHCP agent | :-) | UP | neutron-dhcp-agent | | L3 agent | :-) | UP | neutron-l3-agent | | Metadata agent | XXX | UP | neutron-metadata-agent | +--------------------+-------+-------+---------------------------+ Neutron metadata agent response log: 2025-06-10 15:18:24.306 7 DEBUG neutron.agent.metadata.agent [None req-49747846-774c-40ad-9fef-0eaaf7ea1705 - - - - - -] Request: GET /latest/meta-data/public-ipv4/ HTTP/1.1 host: 169.254.169.254 user-agent: curl/7.84.0 accept: */* x-neutron-network-id: 03739e2a-f552-43b1-9d6f-43db61d03aaf x-forwarded-for: 10.0.0.59 connection: close handle /var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/agent/metadata/agent.py:163 2025-06-10 15:18:24.309 7 DEBUG oslo_concurrency.lockutils [None req-49747846-774c-40ad-9fef-0eaaf7ea1705 - - - - - -] Acquiring lock "oslo_read_shm_alex-aio-epoxy-rocky-7.novalocal_neutron-metadata-agent" by "oslo_messaging._drivers.amqpdriver.QManager.get.<locals>.read_from_shm" inner /var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_concurrency/lockutils.py:405 2025-06-10 15:18:24.309 7 DEBUG oslo_concurrency.lockutils [None req-49747846-774c-40ad-9fef-0eaaf7ea1705 - - - - - -] Lock "oslo_read_shm_alex-aio-epoxy-rocky-7.novalocal_neutron-metadata-agent" acquired by "oslo_messaging._drivers.amqpdriver.QManager.get.<locals>.read_from_shm" :: waited 0.001s inner /var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_concurrency/lockutils.py:410 2025-06-10 15:18:24.311 7 DEBUG oslo_concurrency.lockutils [None req-49747846-774c-40ad-9fef-0eaaf7ea1705 - - - - - -] Lock "oslo_read_shm_alex-aio-epoxy-rocky-7.novalocal_neutron-metadata-agent" "released" by "oslo_messaging._drivers.amqpdriver.QManager.get.<locals>.read_from_shm" :: held 0.001s inner /var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_concurrency/lockutils.py:424 2025-06-10 15:18:24.833 7 DEBUG neutron.agent.metadata.agent [None req-49747846-774c-40ad-9fef-0eaaf7ea1705 - - - - - -] Got ports for remote_address 10.0.0.59, network_id 03739e2a-f552-43b1-9d6f-43db61d03aaf, remote_mac None, router_id None[{'id': 'cdbb6292-e301-4f5c-bc34-33e139eb97fb', 'name': '', 'network_id': '03739e2a-f552-43b1-9d6f-43db61d03aaf', 'tenant_id': '961e2027d2ac49d8b7e76065b42206ac', 'mac_address': 'fa:16:3e:d5:a3:a6', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': 'b68cc942-cfc2-4ee7-be20-5ce5e1f62353', 'device_owner': 'compute:nova', 'standard_attr_id': 662, 'fixed_ips': [{'subnet_id': 'c0cfe0fc-f3af-4ac1-9e11-a4fbbcd31167', 'ip_address': '10.0.0.59'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': ['f2f888b6-1209-4127-8981-acec08ec4eed'], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'alex-aio-epoxy-rocky-7.novalocal', 'binding:vif_type': 'ovs', 'binding:vif_details': {'connectivity ': 'l2', 'port_filter': True, 'ovs_hybrid_plug': True, 'datapath_type': 'system', 'bridge_name': 'br-int', 'bound_drivers': {'0': 'openvswitch'}}, 'port_security_enabled': True, 'tags': [], 'created_at': '2025-06-10T14:57:42Z', 'updated_at': '2025-06-10T14:57:47Z', 'revision_number': 5, 'project_id': '961e2027d2ac49d8b7e76065b42206ac'}] get_port /var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/agent/metadata/agent.py:255 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2113923/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
