Public bug reported:
I am trying to install https://docs.openstack.org/install-guide
/openstack-services.html#minimal-deployment-for-ussuri on CentOS 8, with
network provider option 1.
For the reproduction steps i followed install-guide deployment-for-ussuri.
Other components with trial's and errors did worked, neutron
linuxbridge-agent keeps looping and returning errors.
Logs from linuxbridge-agent.log in the attachment additionally dhcp
agent log returns similar problem in logs:
2020-07-10 12:21:15.060 68787 DEBUG neutron.agent.dhcp.agent [-] Calling driver
for network: 34dc4390-9448-4eba-8be2-a5c3f4cb94a5 action: enable call_driver
/usr/lib/python3.6/site-packages/neutron/agent/dhcp/agent.py:163
2020-07-10 12:21:15.060 68787 DEBUG neutron.agent.linux.utils [-] Unable to
access /var/lib/neutron/dhcp/34dc4390-9448-4eba-8be2-a5c3f4cb94a5/pid; Error:
[Errno 2] No such file or directory:
'/var/lib/neutron/dhcp/34dc4390-9448-4eba-8be2-a5c3f4cb94a5/pid'
get_value_from_file
/usr/lib/python3.6/site-packages/neutron/agent/linux/utils.py:262
2020-07-10 12:21:15.061 68787 INFO oslo.privsep.daemon [-] Running privsep
helper: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf',
'privsep-helper', '--config-file', '/usr/share/neutron/neutron-dist.conf',
'--config-file', '/etc/neutron/neutron.conf', '--config-file',
'/etc/neutron/dhcp_agent.ini', '--config-dir',
'/etc/neutron/conf.d/neutron-dhcp-agent', '--privsep_context',
'neutron.privileged.default', '--privsep_sock_path',
'/tmp/tmphzark8uo/privsep.sock']
2020-07-10 12:21:16.353 68787 CRITICAL oslo.privsep.daemon [-] privsep helper
command exited non-zero (1)
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent [-] Unable to
enable dhcp for 34dc4390-9448-4eba-8be2-a5c3f4cb94a5.:
oslo_privsep.daemon.FailedToDropPrivileges: privsep helper command exited
non-zero (1)
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent Traceback (most
recent call last):
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python3.6/site-packages/neutron/agent/dhcp/agent.py", line 178, in
call_driver
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent
getattr(driver, action)(**action_kwargs)
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 256, in
enable
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent
common_utils.wait_until_true(self._enable, timeout=300)
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python3.6/site-packages/neutron/common/utils.py", line 703, in
wait_until_true
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent while not
predicate():
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 268, in
_enable
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent interface_name
= self.device_manager.setup(self.network)
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 1652, in
setup
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent
ip_lib.IPWrapper().ensure_namespace(network.namespace)
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 249, in
ensure_namespace
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent if not
self.netns.exists(name):
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 728, in
exists
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent return
network_namespace_exists(name)
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 936, in
network_namespace_exists
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent output =
list_network_namespaces(**kwargs)
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 922, in
list_network_namespaces
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent return
privileged.list_netns(**kwargs)
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 246, in
_wrap
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent self.start()
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 258, in
start
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent channel =
daemon.RootwrapClientChannel(context=self)
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 331, in __init__
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent raise
FailedToDropPrivileges(msg)
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent
oslo_privsep.daemon.FailedToDropPrivileges: privsep helper command exited
non-zero (1)
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent
It tries to get /var/lib/neutron/dhcp/34dc4390-9448-4eba-
8be2-a5c3f4cb94a5/pid but it's not there
[root@34dc4390-9448-4eba-8be2-a5c3f4cb94a5]# pwd
/var/lib/neutron/dhcp/34dc4390-9448-4eba-8be2-a5c3f4cb94a5
[root@34dc4390-9448-4eba-8be2-a5c3f4cb94a5]# ls -a
. ..
nmcli con show
NAME UUID TYPE DEVICE
eno3 ad51daed-db15-4380-b559-ccda28fec65e ethernet eno3
eno4 f4d476b2-edaf-45e6-bbb5-84d97b6b5e2f ethernet eno4
openstack network list
+--------------------------------------+----------+--------------------------------------+
| ID | Name | Subnets
|
+--------------------------------------+----------+--------------------------------------+
| 34dc4390-9448-4eba-8be2-a5c3f4cb94a5 | provider |
26401d1b-38c4-4887-bef3-64b746f86e96 |
+--------------------------------------+----------+--------------------------------------+
neutron agent-list
neutron CLI is deprecated and will be removed in the future. Use openstack CLI
instead.
+--------------------------------------+----------------+-----------------------------+-------------------+-------+----------------+------------------------+
| id | agent_type | host
| availability_zone | alive | admin_state_up | binary |
+--------------------------------------+----------------+-----------------------------+-------------------+-------+----------------+------------------------+
| 1ff1b3be-555c-4ef2-a004-b1299f2a8d3d | DHCP agent | X | nova
| :-) | True | neutron-dhcp-agent |
| ff6f0baf-e507-4df2-8cc3-a3eb4bc064e0 | Metadata agent | X |
| :-) | True | neutron-metadata-agent |
+--------------------------------------+----------------+----
eno4 is interface used for provider it has'nt any ip address attached.
sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-iptables = 1
sysctl net.bridge.bridge-nf-call-ip6tables net.bridge.bridge-nf-call-ip6tables
= 1
journalctl -u neutron-linuxbridge-agent.service -xe --no-pager
Jul 10 12:31:02 X sudo[87036]: neutron : TTY=unknown ; PWD=/ ; USER=root ;
COMMAND=/bin/neutron-rootwrap /etc/neutron/rootwrap.conf privsep-helper
--config-file /usr/share/neutron/neutron-dist.conf --config-file
/etc/neutron/neutron.conf --config-file
/etc/neutron/plugins/ml2/linuxbridge_agent.ini --config-dir
/etc/neutron/conf.d/neutron-linuxbridge-agent --privsep_context
neutron.privileged.default --privsep_sock_path /tmp/tmpj1_5u88q/privsep.sock
Jul 10 12:31:03 X neutron-linuxbridge-agent[87013]: 2020-07-10 12:31:03.535
87040 CRITICAL privsep [-] Unhandled error: PermissionError: [Errno 13]
Permission denied
Jul 10 12:31:03 X neutron-linuxbridge-agent[87013]: 2020-07-10 12:31:03.535
87040 ERROR privsep Traceback (most recent call last):
Jul 10 12:31:03 X neutron-linuxbridge-agent[87013]: 2020-07-10 12:31:03.535
87040 ERROR privsep File "/bin/privsep-helper", line 10, in <module>
Jul 10 12:31:03 X neutron-linuxbridge-agent[87013]: 2020-07-10 12:31:03.535
87040 ERROR privsep sys.exit(helper_main())
Jul 10 12:31:03 X neutron-linuxbridge-agent[87013]: 2020-07-10 12:31:03.535
87040 ERROR privsep File
"/usr/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 536, in
helper_main
Jul 10 12:31:03 X neutron-linuxbridge-agent[87013]: 2020-07-10 12:31:03.535
87040 ERROR privsep sock.connect(cfg.CONF.privsep_sock_path)
Jul 10 12:31:03 X neutron-linuxbridge-agent[87013]: 2020-07-10 12:31:03.535
87040 ERROR privsep PermissionError: [Errno 13] Permission denied
Jul 10 12:31:03 X neutron-linuxbridge-agent[87013]: 2020-07-10 12:31:03.535
87040 ERROR privsep
Jul 10 12:31:03 X systemd[1]: neutron-linuxbridge-agent.service: Main process
exited, code=exited, status=1/FAILURE
Jul 10 12:31:03 X systemd[1]: neutron-linuxbridge-agent.service: Failed with
result 'exit-code'.
Jul 10 12:31:04 X systemd[1]: neutron-linuxbridge-agent.service: Service
RestartSec=100ms expired, scheduling restart
cat /etc/sudoers.d/neutron
Defaults:neutron !requiretty
neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap
/etc/neutron/rootwrap.conf *
neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap-daemon
/etc/neutron/rootwrap.conf
neutron ALL = (root) NOPASSWD: ALL
In sudoers file there is directive #includedir /etc/sudoers.d
OpenStack version: Ussuri
Linux distro: CentOS 8
I am trying to install this controller/compute node on the same bare
metal machine.
** Affects: neutron
Importance: Undecided
Status: New
** Attachment added: "linuxbridge-agent.log"
https://bugs.launchpad.net/bugs/1887147/+attachment/5391267/+files/linuxbridge-agent.log
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1887147
Title:
neutron-linuxbridge-agent looping same as dhcp
Status in neutron:
New
Bug description:
I am trying to install https://docs.openstack.org/install-guide
/openstack-services.html#minimal-deployment-for-ussuri on CentOS 8,
with network provider option 1.
For the reproduction steps i followed install-guide deployment-for-ussuri.
Other components with trial's and errors did worked, neutron
linuxbridge-agent keeps looping and returning errors.
Logs from linuxbridge-agent.log in the attachment additionally dhcp
agent log returns similar problem in logs:
2020-07-10 12:21:15.060 68787 DEBUG neutron.agent.dhcp.agent [-] Calling
driver for network: 34dc4390-9448-4eba-8be2-a5c3f4cb94a5 action: enable
call_driver /usr/lib/python3.6/site-packages/neutron/agent/dhcp/agent.py:163
2020-07-10 12:21:15.060 68787 DEBUG neutron.agent.linux.utils [-] Unable to
access /var/lib/neutron/dhcp/34dc4390-9448-4eba-8be2-a5c3f4cb94a5/pid; Error:
[Errno 2] No such file or directory:
'/var/lib/neutron/dhcp/34dc4390-9448-4eba-8be2-a5c3f4cb94a5/pid'
get_value_from_file
/usr/lib/python3.6/site-packages/neutron/agent/linux/utils.py:262
2020-07-10 12:21:15.061 68787 INFO oslo.privsep.daemon [-] Running privsep
helper: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf',
'privsep-helper', '--config-file', '/usr/share/neutron/neutron-dist.conf',
'--config-file', '/etc/neutron/neutron.conf', '--config-file',
'/etc/neutron/dhcp_agent.ini', '--config-dir',
'/etc/neutron/conf.d/neutron-dhcp-agent', '--privsep_context',
'neutron.privileged.default', '--privsep_sock_path',
'/tmp/tmphzark8uo/privsep.sock']
2020-07-10 12:21:16.353 68787 CRITICAL oslo.privsep.daemon [-] privsep helper
command exited non-zero (1)
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent [-] Unable to
enable dhcp for 34dc4390-9448-4eba-8be2-a5c3f4cb94a5.:
oslo_privsep.daemon.FailedToDropPrivileges: privsep helper command exited
non-zero (1)
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent Traceback (most
recent call last):
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python3.6/site-packages/neutron/agent/dhcp/agent.py", line 178, in
call_driver
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent
getattr(driver, action)(**action_kwargs)
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 256, in
enable
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent
common_utils.wait_until_true(self._enable, timeout=300)
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python3.6/site-packages/neutron/common/utils.py", line 703, in
wait_until_true
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent while not
predicate():
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 268, in
_enable
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent
interface_name = self.device_manager.setup(self.network)
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 1652, in
setup
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent
ip_lib.IPWrapper().ensure_namespace(network.namespace)
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 249, in
ensure_namespace
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent if not
self.netns.exists(name):
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 728, in
exists
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent return
network_namespace_exists(name)
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 936, in
network_namespace_exists
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent output =
list_network_namespaces(**kwargs)
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 922, in
list_network_namespaces
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent return
privileged.list_netns(**kwargs)
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 246, in
_wrap
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent self.start()
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 258, in
start
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent channel =
daemon.RootwrapClientChannel(context=self)
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 331, in __init__
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent raise
FailedToDropPrivileges(msg)
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent
oslo_privsep.daemon.FailedToDropPrivileges: privsep helper command exited
non-zero (1)
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent
It tries to get /var/lib/neutron/dhcp/34dc4390-9448-4eba-
8be2-a5c3f4cb94a5/pid but it's not there
[root@34dc4390-9448-4eba-8be2-a5c3f4cb94a5]# pwd
/var/lib/neutron/dhcp/34dc4390-9448-4eba-8be2-a5c3f4cb94a5
[root@34dc4390-9448-4eba-8be2-a5c3f4cb94a5]# ls -a
. ..
nmcli con show
NAME UUID TYPE DEVICE
eno3 ad51daed-db15-4380-b559-ccda28fec65e ethernet eno3
eno4 f4d476b2-edaf-45e6-bbb5-84d97b6b5e2f ethernet eno4
openstack network list
+--------------------------------------+----------+--------------------------------------+
| ID | Name | Subnets
|
+--------------------------------------+----------+--------------------------------------+
| 34dc4390-9448-4eba-8be2-a5c3f4cb94a5 | provider |
26401d1b-38c4-4887-bef3-64b746f86e96 |
+--------------------------------------+----------+--------------------------------------+
neutron agent-list
neutron CLI is deprecated and will be removed in the future. Use openstack
CLI instead.
+--------------------------------------+----------------+-----------------------------+-------------------+-------+----------------+------------------------+
| id | agent_type | host
| availability_zone | alive | admin_state_up | binary |
+--------------------------------------+----------------+-----------------------------+-------------------+-------+----------------+------------------------+
| 1ff1b3be-555c-4ef2-a004-b1299f2a8d3d | DHCP agent | X | nova
| :-) | True | neutron-dhcp-agent |
| ff6f0baf-e507-4df2-8cc3-a3eb4bc064e0 | Metadata agent | X |
| :-) | True | neutron-metadata-agent |
+--------------------------------------+----------------+----
eno4 is interface used for provider it has'nt any ip address attached.
sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-iptables
= 1
sysctl net.bridge.bridge-nf-call-ip6tables
net.bridge.bridge-nf-call-ip6tables = 1
journalctl -u neutron-linuxbridge-agent.service -xe --no-pager
Jul 10 12:31:02 X sudo[87036]: neutron : TTY=unknown ; PWD=/ ; USER=root ;
COMMAND=/bin/neutron-rootwrap /etc/neutron/rootwrap.conf privsep-helper
--config-file /usr/share/neutron/neutron-dist.conf --config-file
/etc/neutron/neutron.conf --config-file
/etc/neutron/plugins/ml2/linuxbridge_agent.ini --config-dir
/etc/neutron/conf.d/neutron-linuxbridge-agent --privsep_context
neutron.privileged.default --privsep_sock_path /tmp/tmpj1_5u88q/privsep.sock
Jul 10 12:31:03 X neutron-linuxbridge-agent[87013]: 2020-07-10 12:31:03.535
87040 CRITICAL privsep [-] Unhandled error: PermissionError: [Errno 13]
Permission denied
Jul 10 12:31:03 X neutron-linuxbridge-agent[87013]: 2020-07-10 12:31:03.535
87040 ERROR privsep Traceback (most recent call last):
Jul 10 12:31:03 X neutron-linuxbridge-agent[87013]: 2020-07-10 12:31:03.535
87040 ERROR privsep File "/bin/privsep-helper", line 10, in <module>
Jul 10 12:31:03 X neutron-linuxbridge-agent[87013]: 2020-07-10 12:31:03.535
87040 ERROR privsep sys.exit(helper_main())
Jul 10 12:31:03 X neutron-linuxbridge-agent[87013]: 2020-07-10 12:31:03.535
87040 ERROR privsep File
"/usr/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 536, in
helper_main
Jul 10 12:31:03 X neutron-linuxbridge-agent[87013]: 2020-07-10 12:31:03.535
87040 ERROR privsep sock.connect(cfg.CONF.privsep_sock_path)
Jul 10 12:31:03 X neutron-linuxbridge-agent[87013]: 2020-07-10 12:31:03.535
87040 ERROR privsep PermissionError: [Errno 13] Permission denied
Jul 10 12:31:03 X neutron-linuxbridge-agent[87013]: 2020-07-10 12:31:03.535
87040 ERROR privsep
Jul 10 12:31:03 X systemd[1]: neutron-linuxbridge-agent.service: Main process
exited, code=exited, status=1/FAILURE
Jul 10 12:31:03 X systemd[1]: neutron-linuxbridge-agent.service: Failed with
result 'exit-code'.
Jul 10 12:31:04 X systemd[1]: neutron-linuxbridge-agent.service: Service
RestartSec=100ms expired, scheduling restart
cat /etc/sudoers.d/neutron
Defaults:neutron !requiretty
neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap
/etc/neutron/rootwrap.conf *
neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap-daemon
/etc/neutron/rootwrap.conf
neutron ALL = (root) NOPASSWD: ALL
In sudoers file there is directive #includedir /etc/sudoers.d
OpenStack version: Ussuri
Linux distro: CentOS 8
I am trying to install this controller/compute node on the same bare
metal machine.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1887147/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
