Reviewed: https://review.openstack.org/370902 Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=35e386f6165a431a066ea92cd317406322053ff7 Submitter: Jenkins Branch: master
commit 35e386f6165a431a066ea92cd317406322053ff7 Author: Brian Haley <[email protected]> Date: Thu Sep 15 10:40:04 2016 -0400 Fix metering-agent iptables restore failure When we removed the trailing space from iptables rules, we broke the code that adds/restores them since it was assuming it could split() based on that space to get the rule. Just detect if the two values are equal, denoting there was no rule instead. This is fine since a self-referencing rule is invalid anyways. Change-Id: I9157ced10de3099790dea7f94aa4e614ebf7f25c Closes-bug: #1623958 ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1623958 Title: Metering-agent fails to restore iptables with 'Bad argument' Status in neutron: Fix Released Bug description: iptables-restore fails with 'Bad argument' when we call 'add_rule' with '' as rule, because IptableManager try to add line below which is invalid format. === -I neutron-meter-l-ba70a353-f3a 1 neutron-meter-l-ba70a353-f3a === This behavior is occurred after the modification. https://github.com/openstack/neutron/commit/5b7c71a327d735134fa0eeb4427d0e1bd1f7d1e5 Created blank by rule is stripped during initializing IptablesRule. https://github.com/openstack/neutron/blob/master/neutron/agent/linux/iptables_manager.py#L124 This stripping blank changes result in _generate_chain_diff_iptables_commands as follows. https://github.com/openstack/neutron/blob/master/neutron/agent/linux/iptables_manager.py#L767 before: '' is set in rule variable. after: chain name(e.g.neutron-meter-l-e648a667-c21) is set as rule variable. Additional string is added and invalid format is generated. Thus, iptable-restore fails. trace in metering-agent.log ============================== 2016-09-15 12:50:39.202 22676 ERROR neutron.agent.linux.utils [req-b3191be2-6daf-42bf-bc3a-e792481da48b da7194b4e98b4c8badc5912bbcd7aea4 9384ef06bc9d4af08a384692c92761c e - - -] Exit code: 2; Stdin: # Generated by iptables_manager *filter :neutron-meter-FORWARD - [0:0] :neutron-meter-INPUT - [0:0] :neutron-meter-OUTPUT - [0:0] :neutron-meter-l-ba70a353-f3a - [0:0] :neutron-meter-local - [0:0] :neutron-meter-r-ba70a353-f3a - [0:0] -I FORWARD 2 -j neutron-meter-FORWARD -I INPUT 1 -j neutron-meter-INPUT -I OUTPUT 2 -j neutron-meter-OUTPUT -I neutron-filter-top 1 -j neutron-meter-local -I neutron-meter-FORWARD 1 -j neutron-meter-r-ba70a353-f3a -I neutron-meter-l-ba70a353-f3a 1 neutron-meter-l-ba70a353-f3a COMMIT # Completed by iptables_manager # Generated by iptables_manager *raw :neutron-meter-OUTPUT - [0:0] :neutron-meter-PREROUTING - [0:0] -I OUTPUT 1 -j neutron-meter-OUTPUT -I PREROUTING 1 -j neutron-meter-PREROUTING COMMIT # Completed by iptables_manager ; Stdout: ; Stderr: Bad argument `neutron-meter-l-ba70a353-f3a' Error occurred at line: 14 Try `iptables-restore -h' or 'iptables-restore --help' for more information. 2016-09-15 12:50:39.204 22676 ERROR neutron.agent.linux.iptables_manager [req-b3191be2-6daf-42bf-bc3a-e792481da48b da7194b4e98b4c8badc5912bbcd7aea4 9384ef06bc9d4af08a384692c92761ce - - -] IPTablesManager.apply failed to apply the following set of iptables rules: 1. # Generated by iptables_manager 2. *filter 3. :neutron-meter-FORWARD - [0:0] 4. :neutron-meter-INPUT - [0:0] 5. :neutron-meter-OUTPUT - [0:0] 6. :neutron-meter-l-ba70a353-f3a - [0:0] 7. :neutron-meter-local - [0:0] 8. :neutron-meter-r-ba70a353-f3a - [0:0] 9. -I FORWARD 2 -j neutron-meter-FORWARD 10. -I INPUT 1 -j neutron-meter-INPUT 11. -I OUTPUT 2 -j neutron-meter-OUTPUT 12. -I neutron-filter-top 1 -j neutron-meter-local 13. -I neutron-meter-FORWARD 1 -j neutron-meter-r-ba70a353-f3a 14. -I neutron-meter-l-ba70a353-f3a 1 neutron-meter-l-ba70a353-f3a 15. COMMIT 16. # Completed by iptables_manager 17. # Generated by iptables_manager 18. *raw 19. :neutron-meter-OUTPUT - [0:0] 20. :neutron-meter-PREROUTING - [0:0] 21. -I OUTPUT 1 -j neutron-meter-OUTPUT 22. -I PREROUTING 1 -j neutron-meter-PREROUTING 23. COMMIT 24. # Completed by iptables_manager 25. 2016-09-15 12:50:39.204 22676 DEBUG oslo_concurrency.lockutils [req-b3191be2-6daf-42bf-bc3a-e792481da48b da7194b4e98b4c8badc5912bbcd7aea4 9384ef06bc9d4af08a384692c92761ce - - -] Releasing semaphore "iptables-qrouter-98b59b73-4490-4834-a02c-ae2e1ea16d64" lock /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:225 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent [req-b3191be2-6daf-42bf-bc3a-e792481da48b da7194b4e98b4c8badc5912bbcd7aea4 9384ef06bc9d4af08a384692c92761ce - - -] Driver neutron.services.metering.drivers.iptables.iptables_driver.IptablesMeteringDriver:add_metering_label runtime error 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent Traceback (most recent call last): 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent File "/opt/stack/neutron/neutron/services/metering/agents/metering_agent.py", line 166, in _invoke_driver 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent return getattr(self.metering_driver, func_name)(context, meterings) 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent File "/usr/local/lib/python2.7/dist-packages/oslo_log/helpers.py", line 48, in wrapper 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent return method(*args, **kwargs) 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent File "/opt/stack/neutron/neutron/services/metering/drivers/iptables/iptables_driver.py", line 262, in add_metering_label 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent self._process_associate_metering_label(router) 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent File "/opt/stack/neutron/neutron/services/metering/drivers/iptables/iptables_driver.py", line 231, in _process_associate_metering_label 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent rm.metering_labels[label_id] = label 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent File "/opt/stack/neutron/neutron/services/metering/drivers/iptables/iptables_driver.py", line 58, in __exit__ 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent self.im.apply() 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent File "/opt/stack/neutron/neutron/agent/linux/iptables_manager.py", line 438, in apply 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent return self._apply() 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent File "/opt/stack/neutron/neutron/agent/linux/iptables_manager.py", line 446, in _apply 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent first = self._apply_synchronized() 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent File "/opt/stack/neutron/neutron/agent/linux/iptables_manager.py", line 532, in _apply_synchronized 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent '\n'.join(log_lines)) 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent self.force_reraise() 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent six.reraise(self.type_, self.value, self.tb) 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent File "/opt/stack/neutron/neutron/agent/linux/iptables_manager.py", line 511, in _apply_synchronized 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent run_as_root=True) 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent File "/opt/stack/neutron/neutron/agent/linux/utils.py", line 138, in execute 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent raise RuntimeError(msg) 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent RuntimeError: Exit code: 2; Stdin: # Generated by iptables_manager 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent *filter 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent :neutron-meter-FORWARD - [0:0] 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent :neutron-meter-INPUT - [0:0] 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent :neutron-meter-OUTPUT - [0:0] 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent :neutron-meter-l-ba70a353-f3a - [0:0] 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent :neutron-meter-local - [0:0] 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent :neutron-meter-r-ba70a353-f3a - [0:0] 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent -I FORWARD 2 -j neutron-meter-FORWARD 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent -I INPUT 1 -j neutron-meter-INPUT 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent -I OUTPUT 2 -j neutron-meter-OUTPUT 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent -I neutron-filter-top 1 -j neutron-meter-local 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent -I neutron-meter-FORWARD 1 -j neutron-meter-r-ba70a353-f3a 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent -I neutron-meter-l-ba70a353-f3a 1 neutron-meter-l-ba70a353-f3a 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent COMMIT 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent # Completed by iptables_manager 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent # Generated by iptables_manager 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent *raw 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent :neutron-meter-OUTPUT - [0:0] 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent :neutron-meter-PREROUTING - [0:0] 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent -I OUTPUT 1 -j neutron-meter-OUTPUT 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent -I PREROUTING 1 -j neutron-meter-PREROUTING 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent COMMIT 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent # Completed by iptables_manager 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent ; Stdout: ; Stderr: Bad argument `neutron-meter-l-ba70a353-f3a' 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent Error occurred at line: 14 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent Try `iptables-restore -h' or 'iptables-restore --help' for more information. 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent 2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent ================================= To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1623958/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
