Public bug reported: It seems that test neutron.tests.fullstack.test_connectivity.TestUninterruptedConnectivityOnL2AgentRestart.test_l2_agent_restart in various LB scenarios (flat, vxlan network) are failing recently pretty often.
Examples of failures: https://09f8e4e92bfb8d2ac89d-b41143eab52d80358d8555f964e9341b.ssl.cf5.rackcdn.com/670611/13/check/neutron-fullstack-with-uwsgi/8f51833/testr_results.html https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_400/790288/1/check/neutron-fullstack-with-uwsgi/40025f9/testr_results.html https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_400/790288/1/check/neutron-fullstack-with-uwsgi/40025f9/testr_results.html https://0603beb4ddbd36de1165-42644bdefd5590a8f7e4e2e8a8a4112f.ssl.cf5.rackcdn.com/787956/1/check/neutron-fullstack-with-uwsgi/7640987/testr_results.html https://e978bdcfc0235dcd9417-6560bc3b6382c1d289b358872777ca09.ssl.cf1.rackcdn.com/787956/1/check/neutron-fullstack-with-uwsgi/779913e/testr_results.html https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_0cb/789648/5/check/neutron-fullstack-with-uwsgi/0cb6d65/testr_results.html Stacktrace: ft1.1: neutron.tests.fullstack.test_connectivity.TestUninterruptedConnectivityOnL2AgentRestart.test_l2_agent_restart(LB,Flat network)testtools.testresult.real._StringException: Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 183, in func return f(self, *args, **kwargs) File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/fullstack/test_connectivity.py", line 236, in test_l2_agent_restart self._assert_ping_during_agents_restart( File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/fullstack/base.py", line 123, in _assert_ping_during_agents_restart common_utils.wait_until_true( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/common/net_helpers.py", line 147, in async_ping f.result() File "/usr/lib/python3.8/concurrent/futures/_base.py", line 432, in result return self.__get_result() File "/usr/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result raise self._exception File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/common/net_helpers.py", line 128, in assert_async_ping ns_ip_wrapper.netns.execute( File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 718, in execute return utils.execute(cmd, check_exit_code=check_exit_code, File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/utils.py", line 156, in execute raise exceptions.ProcessExecutionError(msg, neutron_lib.exceptions.ProcessExecutionError: Exit code: 1; Cmd: ['ip', 'netns', 'exec', 'test-af70cf3a-c531-4fdf-ab4c-31cc69cc2c56', 'ping', '-W', 2, '-c', '1', '20.0.0.212']; Stdin: ; Stdout: PING 20.0.0.212 (20.0.0.212) 56(84) bytes of data. --- 20.0.0.212 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms ; Stderr: I checked linuxbridge-agent logs (2 cases) and I found there error like below: 2021-05-13 15:46:07.721 96421 DEBUG oslo.privsep.daemon [-] privsep: reply[139960964907248]: (4, ()) _call_back /home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.8/site-packages/oslo_privsep/daemon.py:510 2021-05-13 15:46:07.725 96421 DEBUG oslo.privsep.daemon [-] privsep: reply[139960964907248]: (4, None) _call_back /home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.8/site-packages/oslo_privsep/daemon.py:510 2021-05-13 15:46:07.728 96421 DEBUG oslo.privsep.daemon [-] privsep: Exception during request[139960964907248]: Network interface brqa235fa8c-09 not found in namespace None. _process_cmd /home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.8/site-packages/oslo_privsep/daemon.py:488 Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.8/site-packages/oslo_privsep/daemon.py", line 485, in _process_cmd ret = func(*f_args, **f_kwargs) File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.8/site-packages/oslo_privsep/priv_context.py", line 249, in _wrap return func(*args, **kwargs) File "/home/zuul/src/opendev.org/openstack/neutron/neutron/privileged/agent/linux/ip_lib.py", line 278, in delete_ip_address _run_iproute_addr("delete", File "/home/zuul/src/opendev.org/openstack/neutron/neutron/privileged/agent/linux/ip_lib.py", line 239, in _run_iproute_addr idx = get_link_id(device, namespace) File "/home/zuul/src/opendev.org/openstack/neutron/neutron/privileged/agent/linux/ip_lib.py", line 201, in get_link_id raise NetworkInterfaceNotFound(device=device, namespace=namespace) neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface brqa235fa8c-09 not found in namespace None. 2021-05-13 15:46:07.730 96421 DEBUG oslo.privsep.daemon [-] privsep: reply[139960964907248]: (5, 'neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound', ('Network interface brqa235fa8c-09 not found in namespace None.',)) _call_back /home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.8/site-packages/oslo_privsep/daemon.py:510 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server [req-6e40de24-c317-438b-914e-65ea4acea314 - - - - -] Exception during message handling: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface brqa235fa8c-09 not found in namespace None. 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 887, in network_delete 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server self.agent.mgr.delete_bridge(bridge_name) 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 600, in delete_bridge 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server updated = self.update_interface_ip_details(interface, 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 418, in update_interface_ip_details 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server self._update_interface_ip_details(destination, source, ips, 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 410, in _update_interface_ip_details 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server src_device.addr.delete(cidr=ip['cidr']) 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 517, in delete 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server delete_ip_address(cidr, self.name, self._parent.namespace) 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 811, in delete_ip_address 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server privileged.delete_ip_address( 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.8/site-packages/oslo_privsep/priv_context.py", line 247, in _wrap 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server return self.channel.remote_call(name, args, kwargs) 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.8/site-packages/oslo_privsep/daemon.py", line 224, in remote_call 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server raise exc_type(*result[2]) 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface brqa235fa8c-09 not found in namespace None. This may be related to the issue or may be red herring. I don't really know for now. ** Affects: neutron Importance: Critical Status: Confirmed ** Tags: fullstack gate-failure linuxbridge -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1928764 Title: Fullstack test TestUninterruptedConnectivityOnL2AgentRestart failing often with LB agent Status in neutron: Confirmed Bug description: It seems that test neutron.tests.fullstack.test_connectivity.TestUninterruptedConnectivityOnL2AgentRestart.test_l2_agent_restart in various LB scenarios (flat, vxlan network) are failing recently pretty often. Examples of failures: https://09f8e4e92bfb8d2ac89d-b41143eab52d80358d8555f964e9341b.ssl.cf5.rackcdn.com/670611/13/check/neutron-fullstack-with-uwsgi/8f51833/testr_results.html https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_400/790288/1/check/neutron-fullstack-with-uwsgi/40025f9/testr_results.html https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_400/790288/1/check/neutron-fullstack-with-uwsgi/40025f9/testr_results.html https://0603beb4ddbd36de1165-42644bdefd5590a8f7e4e2e8a8a4112f.ssl.cf5.rackcdn.com/787956/1/check/neutron-fullstack-with-uwsgi/7640987/testr_results.html https://e978bdcfc0235dcd9417-6560bc3b6382c1d289b358872777ca09.ssl.cf1.rackcdn.com/787956/1/check/neutron-fullstack-with-uwsgi/779913e/testr_results.html https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_0cb/789648/5/check/neutron-fullstack-with-uwsgi/0cb6d65/testr_results.html Stacktrace: ft1.1: neutron.tests.fullstack.test_connectivity.TestUninterruptedConnectivityOnL2AgentRestart.test_l2_agent_restart(LB,Flat network)testtools.testresult.real._StringException: Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 183, in func return f(self, *args, **kwargs) File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/fullstack/test_connectivity.py", line 236, in test_l2_agent_restart self._assert_ping_during_agents_restart( File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/fullstack/base.py", line 123, in _assert_ping_during_agents_restart common_utils.wait_until_true( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/common/net_helpers.py", line 147, in async_ping f.result() File "/usr/lib/python3.8/concurrent/futures/_base.py", line 432, in result return self.__get_result() File "/usr/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result raise self._exception File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/common/net_helpers.py", line 128, in assert_async_ping ns_ip_wrapper.netns.execute( File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 718, in execute return utils.execute(cmd, check_exit_code=check_exit_code, File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/utils.py", line 156, in execute raise exceptions.ProcessExecutionError(msg, neutron_lib.exceptions.ProcessExecutionError: Exit code: 1; Cmd: ['ip', 'netns', 'exec', 'test-af70cf3a-c531-4fdf-ab4c-31cc69cc2c56', 'ping', '-W', 2, '-c', '1', '20.0.0.212']; Stdin: ; Stdout: PING 20.0.0.212 (20.0.0.212) 56(84) bytes of data. --- 20.0.0.212 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms ; Stderr: I checked linuxbridge-agent logs (2 cases) and I found there error like below: 2021-05-13 15:46:07.721 96421 DEBUG oslo.privsep.daemon [-] privsep: reply[139960964907248]: (4, ()) _call_back /home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.8/site-packages/oslo_privsep/daemon.py:510 2021-05-13 15:46:07.725 96421 DEBUG oslo.privsep.daemon [-] privsep: reply[139960964907248]: (4, None) _call_back /home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.8/site-packages/oslo_privsep/daemon.py:510 2021-05-13 15:46:07.728 96421 DEBUG oslo.privsep.daemon [-] privsep: Exception during request[139960964907248]: Network interface brqa235fa8c-09 not found in namespace None. _process_cmd /home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.8/site-packages/oslo_privsep/daemon.py:488 Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.8/site-packages/oslo_privsep/daemon.py", line 485, in _process_cmd ret = func(*f_args, **f_kwargs) File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.8/site-packages/oslo_privsep/priv_context.py", line 249, in _wrap return func(*args, **kwargs) File "/home/zuul/src/opendev.org/openstack/neutron/neutron/privileged/agent/linux/ip_lib.py", line 278, in delete_ip_address _run_iproute_addr("delete", File "/home/zuul/src/opendev.org/openstack/neutron/neutron/privileged/agent/linux/ip_lib.py", line 239, in _run_iproute_addr idx = get_link_id(device, namespace) File "/home/zuul/src/opendev.org/openstack/neutron/neutron/privileged/agent/linux/ip_lib.py", line 201, in get_link_id raise NetworkInterfaceNotFound(device=device, namespace=namespace) neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface brqa235fa8c-09 not found in namespace None. 2021-05-13 15:46:07.730 96421 DEBUG oslo.privsep.daemon [-] privsep: reply[139960964907248]: (5, 'neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound', ('Network interface brqa235fa8c-09 not found in namespace None.',)) _call_back /home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.8/site-packages/oslo_privsep/daemon.py:510 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server [req-6e40de24-c317-438b-914e-65ea4acea314 - - - - -] Exception during message handling: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface brqa235fa8c-09 not found in namespace None. 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 887, in network_delete 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server self.agent.mgr.delete_bridge(bridge_name) 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 600, in delete_bridge 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server updated = self.update_interface_ip_details(interface, 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 418, in update_interface_ip_details 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server self._update_interface_ip_details(destination, source, ips, 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 410, in _update_interface_ip_details 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server src_device.addr.delete(cidr=ip['cidr']) 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 517, in delete 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server delete_ip_address(cidr, self.name, self._parent.namespace) 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 811, in delete_ip_address 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server privileged.delete_ip_address( 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.8/site-packages/oslo_privsep/priv_context.py", line 247, in _wrap 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server return self.channel.remote_call(name, args, kwargs) 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.8/site-packages/oslo_privsep/daemon.py", line 224, in remote_call 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server raise exc_type(*result[2]) 2021-05-13 15:46:07.730 96075 ERROR oslo_messaging.rpc.server neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface brqa235fa8c-09 not found in namespace None. This may be related to the issue or may be red herring. I don't really know for now. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1928764/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
