Public bug reported: In case when during spawning of the setup of dhcp agent, there will be iptabls error "Another app is currently holding the xtables lock" tests like neutron.tests.functional.agent.test_dhcp_agent.DHCPAgentOVSTestCase.test_enable_isolated_metadata_for_subnet_create_delete and neutron.tests.functional.agent.test_dhcp_agent.DHCPAgentOVSTestCase.test_force_metadata_for_subnet_create_delete may fail with error like below:
2021-03-30 20:57:01.829 61168 DEBUG neutron.agent.linux.dhcp [req-1d14cf38-d8a8-4f3a-858d-4ab6e9b888da - - - - -] Previous DHCP port information: <MagicMock name='DhcpPluginApi().create_dhcp_port()' id='139833378414448'>. Updated DHCP port information: <Mock name='DhcpPluginApi().get_dhcp_port()' id='139833360938464'>. _check_dhcp_port_subnet /home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/dhcp.py:1582 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent [req-1d14cf38-d8a8-4f3a-858d-4ab6e9b888da - - - - -] Unable to enable dhcp for 24e1cf2a-a60d-41a9-9666-a38a90117cf9.: TypeError: can not serialize 'MagicMock' object 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent Traceback (most recent call last): 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/dhcp/agent.py", line 227, in call_driver 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent rv = getattr(driver, action)(**action_kwargs) 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/dhcp.py", line 266, in enable 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent common_utils.wait_until_true(self._enable, timeout=300) 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 707, in wait_until_true 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent while not predicate(): 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/dhcp.py", line 278, in _enable 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent interface_name = self.device_manager.setup(self.network) 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/dhcp.py", line 1692, in setup 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent if ip_lib.ensure_device_is_ready(interface_name, 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 963, in ensure_device_is_ready 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent if not dev.link.exists or not dev.link.address: 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 500, in exists 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent return privileged.interface_exists(self.name, self._parent.namespace) 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_privsep/priv_context.py", line 247, in _wrap 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent return self.channel.remote_call(name, args, kwargs) 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_privsep/daemon.py", line 214, in remote_call 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent result = self.send_recv((Message.CALL.value, name, args, kwargs)) 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_privsep/comm.py", line 170, in send_recv 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent self.writer.send((myid, msg)) 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_privsep/comm.py", line 54, in send 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent buf = msgpack.packb(msg, use_bin_type=True, 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/msgpack/__init__.py", line 35, in packb 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent return Packer(**kwargs).pack(o) 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "msgpack/_packer.pyx", line 298, in msgpack._cmsgpack.Packer.pack 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "msgpack/_packer.pyx", line 295, in msgpack._cmsgpack.Packer.pack 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "msgpack/_packer.pyx", line 264, in msgpack._cmsgpack.Packer._pack 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "msgpack/_packer.pyx", line 264, in msgpack._cmsgpack.Packer._pack 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "msgpack/_packer.pyx", line 264, in msgpack._cmsgpack.Packer._pack 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer._pack 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent TypeError: can not serialize 'MagicMock' object Example of such error https://e9a79c0822aeb557489f-25bbaed08360ac14bb61b985414d524a.ssl.cf2.rackcdn.com/783743/1/check/neutron-functional-with-uwsgi/6f0aa76/controller/logs/dsvm-functional-logs/neutron.tests.functional.agent.test_dhcp_agent.DHCPAgentOVSTestCase.test_enable_isolated_metadata_for_subnet_create_delete.txt It happens like that because in that test there is no mock of neutron.agent.linux.dhcp.DeviceManager._update_dhcp_port() method. Workflow of such test is like below: 1. Call DeviceManager.setup() method 2. This method as one of the first steps will call _update_dhcp_port which will replace dhcp port prepared for test with some other mock, 3. During first run "port" variable in DeviceManager.setup() method is correct so all will work fine but if we hit iptables xlock error, setup() method will be called again and 4. Now "port" local variable is already update by _update_dhcp_port method thus test setup() method will fail as interface_name is now wrong. ** Affects: neutron Importance: High Assignee: Slawek Kaplonski (slaweq) Status: Confirmed ** Tags: functional-tests gate-failure l3-ipam-dhcp -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1922684 Title: Functional dhcp agent tests fails to spawn metadata proxy Status in neutron: Confirmed Bug description: In case when during spawning of the setup of dhcp agent, there will be iptabls error "Another app is currently holding the xtables lock" tests like neutron.tests.functional.agent.test_dhcp_agent.DHCPAgentOVSTestCase.test_enable_isolated_metadata_for_subnet_create_delete and neutron.tests.functional.agent.test_dhcp_agent.DHCPAgentOVSTestCase.test_force_metadata_for_subnet_create_delete may fail with error like below: 2021-03-30 20:57:01.829 61168 DEBUG neutron.agent.linux.dhcp [req-1d14cf38-d8a8-4f3a-858d-4ab6e9b888da - - - - -] Previous DHCP port information: <MagicMock name='DhcpPluginApi().create_dhcp_port()' id='139833378414448'>. Updated DHCP port information: <Mock name='DhcpPluginApi().get_dhcp_port()' id='139833360938464'>. _check_dhcp_port_subnet /home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/dhcp.py:1582 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent [req-1d14cf38-d8a8-4f3a-858d-4ab6e9b888da - - - - -] Unable to enable dhcp for 24e1cf2a-a60d-41a9-9666-a38a90117cf9.: TypeError: can not serialize 'MagicMock' object 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent Traceback (most recent call last): 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/dhcp/agent.py", line 227, in call_driver 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent rv = getattr(driver, action)(**action_kwargs) 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/dhcp.py", line 266, in enable 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent common_utils.wait_until_true(self._enable, timeout=300) 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 707, in wait_until_true 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent while not predicate(): 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/dhcp.py", line 278, in _enable 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent interface_name = self.device_manager.setup(self.network) 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/dhcp.py", line 1692, in setup 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent if ip_lib.ensure_device_is_ready(interface_name, 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 963, in ensure_device_is_ready 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent if not dev.link.exists or not dev.link.address: 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 500, in exists 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent return privileged.interface_exists(self.name, self._parent.namespace) 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_privsep/priv_context.py", line 247, in _wrap 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent return self.channel.remote_call(name, args, kwargs) 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_privsep/daemon.py", line 214, in remote_call 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent result = self.send_recv((Message.CALL.value, name, args, kwargs)) 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_privsep/comm.py", line 170, in send_recv 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent self.writer.send((myid, msg)) 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_privsep/comm.py", line 54, in send 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent buf = msgpack.packb(msg, use_bin_type=True, 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/msgpack/__init__.py", line 35, in packb 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent return Packer(**kwargs).pack(o) 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "msgpack/_packer.pyx", line 298, in msgpack._cmsgpack.Packer.pack 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "msgpack/_packer.pyx", line 295, in msgpack._cmsgpack.Packer.pack 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "msgpack/_packer.pyx", line 264, in msgpack._cmsgpack.Packer._pack 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "msgpack/_packer.pyx", line 264, in msgpack._cmsgpack.Packer._pack 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "msgpack/_packer.pyx", line 264, in msgpack._cmsgpack.Packer._pack 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer._pack 2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent TypeError: can not serialize 'MagicMock' object Example of such error https://e9a79c0822aeb557489f-25bbaed08360ac14bb61b985414d524a.ssl.cf2.rackcdn.com/783743/1/check/neutron-functional-with-uwsgi/6f0aa76/controller/logs/dsvm-functional-logs/neutron.tests.functional.agent.test_dhcp_agent.DHCPAgentOVSTestCase.test_enable_isolated_metadata_for_subnet_create_delete.txt It happens like that because in that test there is no mock of neutron.agent.linux.dhcp.DeviceManager._update_dhcp_port() method. Workflow of such test is like below: 1. Call DeviceManager.setup() method 2. This method as one of the first steps will call _update_dhcp_port which will replace dhcp port prepared for test with some other mock, 3. During first run "port" variable in DeviceManager.setup() method is correct so all will work fine but if we hit iptables xlock error, setup() method will be called again and 4. Now "port" local variable is already update by _update_dhcp_port method thus test setup() method will fail as interface_name is now wrong. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1922684/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
