Reviewed: https://review.openstack.org/514547
Committed:
https://git.openstack.org/cgit/openstack/oslo.rootwrap/commit/?id=7711a6ce31c3ffa8249bfeac53a4d9e306a7299f
Submitter: Zuul
Branch: master
commit 7711a6ce31c3ffa8249bfeac53a4d9e306a7299f
Author: IWAMOTO Toshihiro <iwam...@valinux.co.jp>
Date: Tue Oct 24 16:27:13 2017 +0900
Protect rootwrap daemon socket against multiple threads
Wrap the call with eventlet.Semaphore. Simultaneous Client.execute
calls can fail badly. Alternatively, rootwrap daemon connections
could be made every time when Client.execute is called, without
using a semaphore.
Change-Id: Id9d38832c67f2d81d382cda797a48fee943a27f1
Closes-bug: #1654287
** Changed in: oslo.rootwrap
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1654287
Title:
rootwrap daemon may return output of previous command
Status in neutron:
Fix Released
Status in oslo.rootwrap:
Fix Released
Bug description:
The functional test for netns_cleanup has failed in the gate today [0].
Apparently, when trying to get the list of devices
(ip_lib.get_devices() 'find /sys/class/net -maxdepth 1 -type 1 -printf
%f') through rootwrap_daemon, it's getting the output of the previous
command instead ('netstat -nlp'). This causes that the netns_cleanup
module tries to unplug random devices which correspond to the actual
output of the 'netstat' command.
This bug doesn't look related to the test itself but to
rootwrap_daemon? Maybe due to long output to the netstat command?
Relevant part of the log
2017-01-05 12:17:04.609 27615 DEBUG neutron.agent.linux.utils
[req-68eceb29-052a-4c8c-8152-38bbe636cba5 - - - - -] Running command (rootwrap
daemon): ['ip', 'netns', 'exec',
'qrouter-cf2030c6-c924-45bb-b13b-6774d275b394', 'netstat', '-nlp']
execute_rootwrap_daemon neutron/agent/linux/utils.py:108
2017-01-05 12:17:04.613 27615 DEBUG neutron.agent.linux.utils
[req-68eceb29-052a-4c8c-8152-38bbe636cba5 - - - - -] Exit code: 0 execute
neutron/agent/linux/utils.py:149
2017-01-05 12:17:04.614 27615 DEBUG neutron.agent.linux.utils
[req-68eceb29-052a-4c8c-8152-38bbe636cba5 - - - - -] Running command (rootwrap
daemon): ['ip', 'netns', 'exec',
'qrouter-cf2030c6-c924-45bb-b13b-6774d275b394', 'find', '/sys/class/net',
'-maxdepth', '1', '-type', 'l', '-printf', '%f '] execute_rootwrap_daemon
neutron/agent/linux/utils.py:108
2017-01-05 12:17:04.645 27615 DEBUG neutron.agent.ovsdb.native.vlog [-]
[POLLIN] on fd 14 __log_wakeup
/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/ovs/poller.py:202
2017-01-05 12:17:04.686 27615 DEBUG neutron.agent.linux.utils
[req-68eceb29-052a-4c8c-8152-38bbe636cba5 - - - - -] Exit code: 0 execute
neutron/agent/linux/utils.py:149
2017-01-05 12:17:04.688 27615 DEBUG neutron.agent.linux.utils
[req-68eceb29-052a-4c8c-8152-38bbe636cba5 - - - - -] Running command (rootwrap
daemon): ['ip', 'netns', 'exec',
'qrouter-cf2030c6-c924-45bb-b13b-6774d275b394', 'ip', 'link', 'delete',
'Active'] execute_rootwrap_daemon neutron/agent/linux/utils.py:108
2017-01-05 12:17:04.746 27615 DEBUG neutron.agent.linux.utils
[req-68eceb29-052a-4c8c-8152-38bbe636cba5 - - - - -] Exit code: 0 execute
neutron/agent/linux/utils.py:149
2017-01-05 12:17:04.747 27615 DEBUG neutron.agent.linux.utils
[req-68eceb29-052a-4c8c-8152-38bbe636cba5 - - - - -] Running command (rootwrap
daemon): ['ip', 'netns', 'exec',
'qrouter-cf2030c6-c924-45bb-b13b-6774d275b394', 'ip', 'link', 'delete',
'Internet'] execute_rootwrap_daemon neutron/agent/linux/utils.py:108
2017-01-05 12:17:04.758 27615 DEBUG neutron.agent.ovsdb.native.vlog [-]
[POLLIN] on fd 14 __log_wakeup
/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/ovs/poller.py:202
2017-01-05 12:17:04.815 27615 DEBUG neutron.agent.ovsdb.native.vlog [-]
[POLLIN] on fd 14 __log_wakeup
/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/ovs/poller.py:202
2017-01-05 12:17:04.822 27615 DEBUG neutron.agent.ovsdb.native.vlog [-]
[POLLIN] on fd 7 __log_wakeup
/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/ovs/poller.py:202
2017-01-05 12:17:04.822 27615 DEBUG neutron.agent.ovsdb.impl_idl [-] Running
txn command(idx=0): InterfaceToBridgeCommand(name=Internet) do_commit
neutron/agent/ovsdb/impl_idl.py:100
2017-01-05 12:17:04.823 27615 DEBUG neutron.agent.ovsdb.impl_idl [-]
Transaction aborted do_commit neutron/agent/ovsdb/impl_idl.py:124
2017-01-05 12:17:04.824 27615 DEBUG neutron.cmd.netns_cleanup
[req-68eceb29-052a-4c8c-8152-38bbe636cba5 - - - - -] Unable to find bridge for
device: Internet unplug_device neutron/cmd/netns_cleanup.py:138
2017-01-05 12:17:04.824 27615 DEBUG neutron.agent.linux.utils
[req-68eceb29-052a-4c8c-8152-38bbe636cba5 - - - - -] Running command (rootwrap
daemon): ['ip', 'netns', 'exec',
'qrouter-cf2030c6-c924-45bb-b13b-6774d275b394', 'ip', 'link', 'delete',
'connections'] execute_rootwrap_daemon neutron/agent/linux/utils.py:108
....
2017-01-05 12:17:06.388 27615 DEBUG neutron.cmd.netns_cleanup
[req-68eceb29-052a-4c8c-8152-38bbe636cba5 - - - - -] Unable to find bridge for
device: Path unplug_device neutron/cmd/netns_cleanup.py:138
2017-01-05 12:17:06.389 27615 DEBUG neutron.agent.linux.utils
[req-68eceb29-052a-4c8c-8152-38bbe636cba5 - - - - -] Running command (rootwrap
daemon): ['ip', '-o', 'netns', 'list'] execute_rootwrap_daemon
neutron/agent/linux/utils.py:108
2017-01-05 12:17:06.454 27615 ERROR neutron.agent.linux.utils
[req-68eceb29-052a-4c8c-8152-38bbe636cba5 - - - - -] Exit code: 1; Stdin: ;
Stdout: ; Stderr: Cannot find device "Path"
2017-01-05 12:17:06.454 27615 ERROR neutron.cmd.netns_cleanup
[req-68eceb29-052a-4c8c-8152-38bbe636cba5 - - - - -] Error unable to destroy
namespace: qrouter-cf2030c6-c924-45bb-b13b-6774d275b394
2017-01-05 12:17:06.454 27615 ERROR neutron.cmd.netns_cleanup Traceback (most
recent call last):
2017-01-05 12:17:06.454 27615 ERROR neutron.cmd.netns_cleanup File
"neutron/cmd/netns_cleanup.py", line 250, in destroy_namespace
2017-01-05 12:17:06.454 27615 ERROR neutron.cmd.netns_cleanup
ip.garbage_collect_namespace()
2017-01-05 12:17:06.454 27615 ERROR neutron.cmd.netns_cleanup File
"neutron/agent/linux/ip_lib.py", line 222, in garbage_collect_namespace
2017-01-05 12:17:06.454 27615 ERROR neutron.cmd.netns_cleanup if
self.namespace and self.netns.exists(self.namespace):
2017-01-05 12:17:06.454 27615 ERROR neutron.cmd.netns_cleanup File
"neutron/agent/linux/ip_lib.py", line 888, in exists
2017-01-05 12:17:06.454 27615 ERROR neutron.cmd.netns_cleanup
run_as_root=cfg.CONF.AGENT.use_helper_for_ns_read)
2017-01-05 12:17:06.454 27615 ERROR neutron.cmd.netns_cleanup File
"neutron/agent/linux/ip_lib.py", line 107, in _execute
2017-01-05 12:17:06.454 27615 ERROR neutron.cmd.netns_cleanup
log_fail_as_error=log_fail_as_error)
2017-01-05 12:17:06.454 27615 ERROR neutron.cmd.netns_cleanup File
"neutron/agent/linux/utils.py", line 147, in execute
2017-01-05 12:17:06.454 27615 ERROR neutron.cmd.netns_cleanup raise
ProcessExecutionError(msg, returncode=returncode)
2017-01-05 12:17:06.454 27615 ERROR neutron.cmd.netns_cleanup
ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot find
device "Path"
[0] http://logs.openstack.org/51/396651/15/check/gate-neutron-dsvm-
functional-ubuntu-xenial/5d268f0/logs/dsvm-functional-
logs/neutron.tests.functional.cmd.test_netns_cleanup.NetnsCleanupTest.test_cleanup_network_namespaces_cleans_dhcp_and_l3_namespaces.txt.gz
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1654287/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
