Bug closed due to lack of activity, please feel free to reopen if
needed.
** Changed in: neutron
Status: In Progress => Won't Fix
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1848187
Title:
DHCP agent timing out spawning dnsmasq
Status in neutron:
Won't Fix
Bug description:
This is using `stable/stein` deployed with kolla. I'm seeing timeouts
when spawning the DHCP process which seems to result in an
inconsistent state.
2019-10-14 18:33:04.896 6 DEBUG neutron.agent.linux.utils
[req-14bec9b8-4264-401d-bcfd-fb588458a0a8 - - - - -] Running command: ['sudo',
'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec',
'qdhcp-432513ea-ddb7-4400-af94-b0017bb50df7', 'sysctl', '-w',
'net.ipv4.conf.all.promote_secondaries=1'] create_process
/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:87
2019-10-14 18:33:08.981 6 DEBUG neutron.agent.linux.utils
[req-14bec9b8-4264-401d-bcfd-fb588458a0a8 - - - - -] Running command: ['sudo',
'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec',
'qdhcp-432513ea-ddb7-4400-af94-b0017bb50df7', 'sysctl', '-w',
'net.ipv4.ip_nonlocal_bind=1'] create_process
/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:87
2019-10-14 18:33:13.924 6 DEBUG neutron.agent.linux.utils
[req-14bec9b8-4264-401d-bcfd-fb588458a0a8 - - - - -] Running command: ['sudo',
'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec',
'qdhcp-432513ea-ddb7-4400-af94-b0017bb50df7', 'sysctl', '-w',
'net.ipv6.conf.default.accept_ra=0'] create_process
/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:87
2019-10-14 18:33:35.364 6 DEBUG oslo_concurrency.lockutils
[req-14bec9b8-4264-401d-bcfd-fb588458a0a8 - - - - -] Acquired lock
"iptables-qdhcp-432513ea-ddb7-4400-af94-b0017bb50df7" lock
/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:265
2019-10-14 18:33:35.366 6 DEBUG oslo_concurrency.lockutils
[req-14bec9b8-4264-401d-bcfd-fb588458a0a8 - - - - -] Acquired external
semaphore "iptables-qdhcp-432513ea-ddb7-4400-af94-b0017bb50df7" lock
/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:272
2019-10-14 18:33:35.366 6 DEBUG neutron.agent.linux.utils
[req-14bec9b8-4264-401d-bcfd-fb588458a0a8 - - - - -] Running command: ['sudo',
'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec',
'qdhcp-432513ea-ddb7-4400-af94-b0017bb50df7', 'iptables-save'] create_process
/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:87
2019-10-14 18:33:39.431 6 DEBUG neutron.agent.linux.utils
[req-14bec9b8-4264-401d-bcfd-fb588458a0a8 - - - - -] Running command: ['sudo',
'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec',
'qdhcp-432513ea-ddb7-4400-af94-b0017bb50df7', 'iptables-restore', '-n', '-w',
'10', '-W', '200000'] create_process
/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:87
2019-10-14 18:33:48.335 6 DEBUG neutron.agent.linux.utils
[req-14bec9b8-4264-401d-bcfd-fb588458a0a8 - - - - -] Running command: ['sudo',
'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec',
'qdhcp-432513ea-ddb7-4400-af94-b0017bb50df7', 'ip6tables-save'] create_process
/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:87
2019-10-14 18:33:50.865 6 DEBUG neutron.agent.linux.utils
[req-14bec9b8-4264-401d-bcfd-fb588458a0a8 - - - - -] Running command: ['sudo',
'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec',
'qdhcp-432513ea-ddb7-4400-af94-b0017bb50df7', 'ip6tables-restore', '-n', '-w',
'10', '-W', '200000'] create_process
/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:87
2019-10-14 18:33:52.677 6 DEBUG neutron.agent.linux.iptables_manager
[req-14bec9b8-4264-401d-bcfd-fb588458a0a8 - - - - -] IPTablesManager.apply
completed with success. 94 iptables commands were issued _apply_synchronized
/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:620
2019-10-14 18:33:52.678 6 DEBUG oslo_concurrency.lockutils
[req-14bec9b8-4264-401d-bcfd-fb588458a0a8 - - - - -] Releasing lock
"iptables-qdhcp-432513ea-ddb7-4400-af94-b0017bb50df7" lock
/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:281
2019-10-14 18:33:56.488 6 DEBUG neutron.agent.linux.utils
[req-14bec9b8-4264-401d-bcfd-fb588458a0a8 - - - - -] Running command: ['sudo',
'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec',
'qdhcp-432513ea-ddb7-4400-af94-b0017bb50df7', 'ip', '-4', 'route', 'list',
'dev', 'tapc29a1605-1a'] create_process
/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:87
2019-10-14 18:33:59.880 6 DEBUG neutron.agent.linux.dhcp
[req-14bec9b8-4264-401d-bcfd-fb588458a0a8 - - - - -] Setting IPv4 gateway for
dhcp netns on net 432513ea-ddb7-4400-af94-b0017bb50df7 to 10.100.0.1
_set_default_route_ip_version
/usr/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py:1262
2019-10-14 18:33:59.881 6 DEBUG neutron.agent.linux.utils
[req-14bec9b8-4264-401d-bcfd-fb588458a0a8 - - - - -] Running command: ['sudo',
'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec',
'qdhcp-432513ea-ddb7-4400-af94-b0017bb50df7', 'ip', '-4', 'route', 'replace',
'default', 'via', '10.100.0.1', 'dev', 'tapc29a1605-1a'] create_process
/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:87
2019-10-14 18:34:01.536 6 DEBUG neutron.agent.linux.utils
[req-14bec9b8-4264-401d-bcfd-fb588458a0a8 - - - - -] Running command: ['sudo',
'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec',
'qdhcp-432513ea-ddb7-4400-af94-b0017bb50df7', 'ip', '-6', 'route', 'list',
'dev', 'tapc29a1605-1a'] create_process
/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:87
2019-10-14 18:34:02.369 6 ERROR neutron.agent.linux.dhcp
[req-14bec9b8-4264-401d-bcfd-fb588458a0a8 - - - - -] Failed to start DHCP
process for network 432513ea-ddb7-4400-af94-b0017bb50df7: WaitTimeout: Timed
out after 60 seconds
2019-10-14 18:34:02.370 6 DEBUG neutron.agent.linux.utils
[req-14bec9b8-4264-401d-bcfd-fb588458a0a8 - - - - -] Unable to access
/var/lib/neutron/external/pids/432513ea-ddb7-4400-af94-b0017bb50df7.pid.haproxy
get_value_from_file
/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:261
2019-10-14 18:34:02.370 6 DEBUG neutron.agent.metadata.driver
[req-14bec9b8-4264-401d-bcfd-fb588458a0a8 - - - - -] haproxy_cfg =
2019-10-14 18:34:02.371 6 DEBUG neutron.agent.linux.utils
[req-14bec9b8-4264-401d-bcfd-fb588458a0a8 - - - - -] Running command: ['sudo',
'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec',
'qdhcp-432513ea-ddb7-4400-af94-b0017bb50df7', 'haproxy', '-f',
'/var/lib/neutron/ns-metadata-proxy/432513ea-ddb7-4400-af94-b0017bb50df7.conf']
create_process /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:87
2019-10-14 18:34:04.003 6 DEBUG oslo_concurrency.lockutils
[req-14bec9b8-4264-401d-bcfd-fb588458a0a8 - - - - -] Lock "resize_greenpool"
acquired by "neutron.agent.dhcp.agent._resize_process_pool" :: waited 0.000s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327
2019-10-14 18:34:04.003 6 DEBUG oslo_concurrency.lockutils
[req-14bec9b8-4264-401d-bcfd-fb588458a0a8 - - - - -] Lock "resize_greenpool"
released by "neutron.agent.dhcp.agent._resize_process_pool" :: held 0.001s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339
clush: centos@bm-controller[01-02] (2): exited with exit code 1'
Notice that the calls to setup the network namespace are taking a
longer than 60 seconds. It appears we never get past:
https://github.com/openstack/neutron/blob/master/neutron/agent/linux/dhcp.py#L228,
to actually spawn the process and that no retries are attemped.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1848187/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
