** This bug is no longer a duplicate of bug 1930414 Traffic leaked from dhcp port before vlan tag is applied
-- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1953165 Title: DHCP agent fails to fully configure DHCP namespaces because of duplicate address detected Status in neutron: Confirmed Bug description: After upgrading a Neutron/ML2 OVS deployment from Ussuri to Victoria, updating the host OS from CentOS Linux 8 to CentOS Stream 8, and rebooting, DHCP was not functional on some but not all networks. DHCP agent logs included the following error multiple times: 2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent [-] Failure waiting for address fe80::a9fe:a9fe to become ready: Duplicate address detected: neutron.agent.linux.ip_lib.AddressNotReady: Failure waiting for address fe80::a9fe:a9fe to become ready: Duplicate address detected 2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent Traceback (most recent call last): 2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/common/utils.py", line 165, in call 2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent return func(*args, **kwargs) 2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/dhcp/agent.py", line 401, in safe_configure_dhcp_for_network 2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent self.configure_dhcp_for_network(network) 2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/osprofiler/profiler.py", line 160, in wrapper 2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent result = f(*args, **kwargs) 2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/dhcp/agent.py", line 415, in configure_dhcp_for_network 2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent self.update_isolated_metadata_proxy(network) 2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/osprofiler/profiler.py", line 160, in wrapper 2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent result = f(*args, **kwargs) 2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/dhcp/agent.py", line 758, in update_isolated_metadata_proxy 2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent self.enable_isolated_metadata_proxy(network) 2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/osprofiler/profiler.py", line 160, in wrapper 2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent result = f(*args, **kwargs) 2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/dhcp/agent.py", line 816, in enable_isolated_metadata_proxy 2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent self.conf, bind_address=constants.METADATA_V4_IP, **kwargs) 2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/metadata/driver.py", line 271, in spawn_monitored_metadata_proxy 2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent ).wait_until_address_ready(address=bind_address_v6) 2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 597, in wait_until_address_ready 2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent exception=AddressNotReady(address=address, reason=errmsg)) 2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/common/utils.py", line 701, in wait_until_true 2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent while not predicate(): 2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 591, in is_address_ready 2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent address=address, reason=_('Duplicate address detected')) 2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent neutron.agent.linux.ip_lib.AddressNotReady: Failure waiting for address fe80::a9fe:a9fe to become ready: Duplicate address detected 2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent The tap interface inside each affected qdhcp namespace was in a state like this: 35: tap0f8bb343-c1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:ed:6f:60 brd ff:ff:ff:ff:ff:ff inet 169.254.169.254/32 brd 169.254.169.254 scope global tap0f8bb343-c1 valid_lft forever preferred_lft forever inet 10.18.0.10/16 brd 10.18.255.255 scope global tap0f8bb343-c1 valid_lft forever preferred_lft forever inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:feed:6f60/64 scope link valid_lft forever preferred_lft forever Note the dadfailed status on the fe80::a9fe:a9fe/64 address, which caused Neutron to raise an AddressNotReady exception. I tried restarting dhcp-agent multiple times. Occasionally DHCP for one network would configure correctly, but most of the times the list of affected networks would stay the same. I found out that removing the fe80::a9fe:a9fe/64 address from the tap interface of each affected namespace followed by restarting dhcp-agent fixed the issue: no more dadfailed status. Version information: * OpenStack Victoria deployed with Kolla source images * neutron 17.2.2.dev70 (using stable/victoria from 2021-11-28) * CentOS Stream release 8 * Linux kernel 4.18.0-348.2.1.el8_5.x86_64 #1 SMP Tue Nov 16 14:42:35 UTC 2021 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1953165/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
