Reviewed: https://review.openstack.org/315389 Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7962dd49eff154acc5d69a6e2a59322a0b2a623b Submitter: Jenkins Branch: master
commit 7962dd49eff154acc5d69a6e2a59322a0b2a623b Author: Hunt Xu <[email protected]> Date: Thu May 12 15:05:24 2016 +0800 Metering: sync only active routers hosted on the same host When syncing data from neutron server, metering-agent may receive information about routers that are not hosted by the l3-agent on the same host, because the server didn't filter them out. This could lead to the following problems: * metering-agent tries to setup iptables rules for a router that is not on the host * metering-agent tries to get get traffic counters for a router that was once on the host but is already removed * metering-agent not sets up iptables rules for a router that is removed then added back to the host, because nothing about the router is changed from metering-agent's perspective This commit fixes the aforementioned problems by making metering-agent only receive information about routers that are on the same host, and update metering-agent's knowledge about which routers it should care. However, there could still be problem if one removes then adds a router back to the same l3-agent, or just sets the router's admin_state_up property to False then True in a short time(shorter than the interval between two syncs). Because the metering-agent sees nothing changed while during the same time the router's namespace is removed and added back on the host. Thus metering-agent will fail to get such router's traffic counters. This commit also make iptables-driver to forget such routers and leave the metering-agent to reconfigure them later. Closes-Bug: #1580548 Change-Id: Ia6ae82c676582b06710d6f96b9938c215258182d Signed-off-by: Hunt Xu <[email protected]> ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1580548 Title: Metering-agent's routers information is inconsistent with neutron- server Status in neutron: Fix Released Bug description: With the current implement, under certain curcumstance, router information(including internal-stored data, iptables settings) kept and set up by metering-agent could be inconsistent with what the neutron-server suppose it to be. And that cause the metering-agent to run into a problematic status. Version: Neutron master commit 7327e8c21(Merge "Fix update target tenant RBAC external path") Steps to reproduce: 1. Create internal net, subnet, router. Set external gateway for router, add interface to the router for the created subnet. 2. Create neutron metering label and rule. 3. Make sure the created router is added to an active l3-agent. 4. Create another router with --admin_state_up=False, and set its external gateway 5. Use 'neutron l3-agent-router-remove' to remove the first router from the l3 agent. 6. Use 'neutron l3-agent-router-add' to add the first router back to the l3 agent by which it was hosted. (In the following logs, the first created router's id is 0dd35572 -735a-4ecd-8429-e047ed385100, the second one is 392820a7-3908-49eb- 98a3-967f1638432f) Logs in metering-agent.log after step 4: 2016-05-11 18:32:49.915 26409 DEBUG neutron.agent.linux.utils [req-0515c6f8-cbd1-47d4-a45e-f89e6b6e97b4 - - - - -] Running command: ['sudo', 'ip', 'netns', 'exec', 'qrouter-392820a7-3908-49eb-98a3-967f1638432f', 'iptables-save'] create_process /usr/local/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:82 2016-05-11 18:32:49.926 26409 ERROR neutron.agent.linux.utils [req-0515c6f8-cbd1-47d4-a45e-f89e6b6e97b4 - - - - -] Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-392820a7-3908-49eb-98a3-967f1638432f": No such file or directory 2016-05-11 18:32:49.926 26409 DEBUG oslo_concurrency.lockutils [req-0515c6f8-cbd1-47d4-a45e-f89e6b6e97b4 - - - - -] Releasing semaphore "iptables-qrouter-392820a7-3908-49eb-98a3-967f1638432f" lock /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:225 2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent [req-0515c6f8-cbd1-47d4-a45e-f89e6b6e97b4 - - - - -] Driver neutron.services.metering.drivers.iptables.iptables_driver.IptablesMeteringDriver:update_routers runtime error 2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent Traceback (most recent call last): 2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent File "/usr/local/lib/python2.7/dist-packages/neutron/services/metering/agents/metering_agent.py", line 177, in _invoke_driver 2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent return getattr(self.metering_driver, func_name)(context, meterings) 2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent File "/usr/local/lib/python2.7/dist-packages/oslo_log/helpers.py", line 48, in wrapper 2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent return method(*args, **kwargs) 2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent File "/usr/local/lib/python2.7/dist-packages/neutron/services/metering/drivers/iptables/iptables_driver.py", line 125, in update_routers 2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent self._process_associate_metering_label(router) 2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent File "/usr/local/lib/python2.7/dist-packages/neutron/services/metering/drivers/iptables/iptables_driver.py", line 228, in _process_associate_metering_label 2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent rm.metering_labels[label_id] = label 2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent File "/usr/local/lib/python2.7/dist-packages/neutron/services/metering/drivers/iptables/iptables_driver.py", line 58, in __exit__ 2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent self.im.apply() 2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/iptables_manager.py", line 431, in apply 2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent return self._apply() 2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/iptables_manager.py", line 439, in _apply 2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent return self._apply_synchronized() 2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/iptables_manager.py", line 465, in _apply_synchronized 2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent save_output = self.execute(args, run_as_root=True) 2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 137, in execute 2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent raise RuntimeError(msg) 2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-392820a7-3908-49eb-98a3-967f1638432f": No such file or directory 2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent 2016-05-11 18:32:49.927 26409 ERROR neutron.services.metering.agents.metering_agent ...Skipping... 2016-05-11 18:33:06.776 26409 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'ip', 'netns', 'exec', 'qrouter-392820a7-3908-49eb-98a3-967f1638432f', 'iptables', '-t', 'filter', '-L', 'neutron-meter-l-1811c9b9-f9b', '-n', '-v', '-x', '-Z'] create_process /usr/local/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:82 2016-05-11 18:33:06.787 26409 ERROR neutron.agent.linux.utils [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-392820a7-3908-49eb-98a3-967f1638432f": No such file or directory 2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver [-] Failed to get traffic counters, router: {u'status': u'ACTIVE', u'name': u'router-two', u'gw_port_id': u'729fba34-4aec-415d-bef8-b93b34e69536', u'admin_state_up': False, u'tenant_id': u'e60fad96858e431cb3da4f0c1ab2d733', u'_metering_labels': [{u'rules': [{u'remote_ip_prefix': u'192.168.88.0/24', u'direction': u'egress', u'metering_label_id': u'1811c9b9-f9b0-40a6-839a-250b20674860', u'id': u'007e3f9e-4e7c-439d-a606-db6ad0ae11ca', u'excluded': False}], u'id': u'1811c9b9-f9b0-40a6-839a-250b20674860'}, {u'rules': [{u'remote_ip_prefix': u'192.168.88.0/24', u'direction': u'ingress', u'metering_label_id': u'd05ce8fb-c013-4ae4-bac5-de1be3f65331', u'id': u'6c580a70-5dec-4225-ade8-44e03fdc9f1b', u'excluded': False}], u'id': u'd05ce8fb-c013-4ae4-bac5-de1be3f65331'}], u'id': u'392820a7-3908-49eb-98a3-967f1638432f'} 2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver Traceback (most recent call last): 2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver File "/usr/local/lib/python2.7/dist-packages/neutron/services/metering/drivers/iptables/iptables_driver.py", line 359, in get_traffic_counters 2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver chain, wrap=False, zero=True) 2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/iptables_manager.py", line 669, in get_traffic_counters 2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver current_table = self.execute(args, run_as_root=True) 2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 137, in execute 2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver raise RuntimeError(msg) 2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-392820a7-3908-49eb-98a3-967f1638432f": No such file or directory 2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver 2016-05-11 18:33:06.788 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver 2016-05-11 18:33:06.790 26409 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'ip', 'netns', 'exec', 'qrouter-392820a7-3908-49eb-98a3-967f1638432f', 'iptables', '-t', 'filter', '-L', 'neutron-meter-l-d05ce8fb-c01', '-n', '-v', '-x', '-Z'] create_process /usr/local/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:82 Logs in metering-agent.log after step 5: 2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver [-] Failed to get traffic counters, router: {u'status': u'ACTIVE', u'name': u'router-two', u'gw_port_id': u'729fba34-4aec-415d-bef8-b93b34e69536', u'admin_state_up': False, u'tenant_id': u'e60fad96858e431cb3da4f0c1ab2d733', u'_metering_labels': [{u'rules': [{u'remote_ip_prefix': u'192.168.88.0/24', u'direction': u'egress', u'metering_label_id': u'1811c9b9-f9b0-40a6-839a-250b20674860', u'id': u'007e3f9e-4e7c-439d-a606-db6ad0ae11ca', u'excluded': False}], u'id': u'1811c9b9-f9b0-40a6-839a-250b20674860'}, {u'rules': [{u'remote_ip_prefix': u'192.168.88.0/24', u'direction': u'ingress', u'metering_label_id': u'd05ce8fb-c013-4ae4-bac5-de1be3f65331', u'id': u'6c580a70-5dec-4225-ade8-44e03fdc9f1b', u'excluded': False}], u'id': u'd05ce8fb-c013-4ae4-bac5-de1be3f65331'}], u'id': u'392820a7-3908-49eb-98a3-967f1638432f'} 2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver Traceback (most recent call last): 2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver File "/usr/local/lib/python2.7/dist-packages/neutron/services/metering/drivers/iptables/iptables_driver.py", line 359, in get_traffic_counters 2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver chain, wrap=False, zero=True) 2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/iptables_manager.py", line 669, in get_traffic_counters 2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver current_table = self.execute(args, run_as_root=True) 2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 137, in execute 2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver raise RuntimeError(msg) 2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-392820a7-3908-49eb-98a3-967f1638432f": No such file or directory 2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver 2016-05-11 18:33:36.805 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver 2016-05-11 18:33:36.807 26409 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'ip', 'netns', 'exec', 'qrouter-0dd35572-735a-4ecd-8429-e047ed385100', 'iptables', '-t', 'filter', '-L', 'neutron-meter-l-1811c9b9-f9b', '-n', '-v', '-x', '-Z'] create_process /usr/local/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:82 2016-05-11 18:33:36.818 26409 ERROR neutron.agent.linux.utils [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-0dd35572-735a-4ecd-8429-e047ed385100": No such file or directory Logs in metering-agent.log after step 6: 2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver [-] Failed to get traffic counters, router: {u'status': u'ACTIVE', u'name': u'router-one', u'gw_port_id': u'a4137128-27d6-43d6-b04d-a5df8ea62b02', u'admin_state_up': True, u'tenant_id': u'e60fad96858e431cb3da4f0c1ab2d733', u'_metering_labels': [{u'rules': [{u'remote_ip_prefix': u'192.168.88.0/24', u'direction': u'egress', u'metering_label_id': u'1811c9b9-f9b0-40a6-839a-250b20674860', u'id': u'007e3f9e-4e7c-439d-a606-db6ad0ae11ca', u'excluded': False}], u'id': u'1811c9b9-f9b0-40a6-839a-250b20674860'}, {u'rules': [{u'remote_ip_prefix': u'192.168.88.0/24', u'direction': u'ingress', u'metering_label_id': u'd05ce8fb-c013-4ae4-bac5-de1be3f65331', u'id': u'6c580a70-5dec-4225-ade8-44e03fdc9f1b', u'excluded': False}], u'id': u'd05ce8fb-c013-4ae4-bac5-de1be3f65331'}], u'id': u'0dd35572-735a-4ecd-8429-e047ed385100'} 2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver Traceback (most recent call last): 2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver File "/usr/local/lib/python2.7/dist-packages/neutron/services/metering/drivers/iptables/iptables_driver.py", line 359, in get_traffic_counters 2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver chain, wrap=False, zero=True) 2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/iptables_manager.py", line 669, in get_traffic_counters 2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver current_table = self.execute(args, run_as_root=True) 2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 137, in execute 2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver raise RuntimeError(msg) 2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: iptables: No chain/target/match by that name. 2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver 2016-05-11 18:34:06.856 26409 ERROR neutron.services.metering.drivers.iptables.iptables_driver 2016-05-11 18:34:06.858 26409 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'ip', 'netns', 'exec', 'qrouter-0dd35572-735a-4ecd-8429-e047ed385100', 'iptables', '-t', 'filter', '-L', 'neutron-meter-l-d05ce8fb-c01', '-n', '-v', '-x', '-Z'] create_process /usr/local/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:82 2016-05-11 18:34:06.911 26409 ERROR neutron.agent.linux.utils [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: iptables: No chain/target/match by that name. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1580548/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
