Hi,
I'm inclined to agree with Oleg here. If 2 of 3 rabbit servers are
working fine and agent is still connected to the broken one, I think
it's something for oslo.messaging. Neutron is just using oslo.messaging
project to connect with rabbitmq.
** Changed in: neutron
Status: New => Incomplete
** Also affects: oslo.messaging
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1873265
Title:
VM can't be accessed, openflow rules are removed and neutron-ovs-agent
is always reporting errors after rabbitmq is in split-brain state
Status in neutron:
Incomplete
Status in oslo.messaging:
New
Bug description:
After rabbitmq cluster is in split brain state because of some network
issues, neutron ovs agents in compute nodes are always reporting error
info, some open flow rules are removed, so VMs can't be accessed
normally.
Here are some important error info:
2020-04-16 08:35:05.578 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Failed
reporting state!: MessagingTimeout: Timed out waiting for a reply to message ID
ede4a709807e43dc90df4469ca9d88f3
2020-04-16 08:35:05.578 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most
recent call last):
2020-04-16 08:35:05.578 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 328, in _report_state
2020-04-16 08:35:05.578 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent True)
2020-04-16 08:35:05.578 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/rpc.py",
line 97, in report_state
2020-04-16 08:35:05.578 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return
method(context, 'report_state', **kwargs)
2020-04-16 08:35:05.578 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py",
line 179, in call
2020-04-16 08:35:05.578 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
retry=self.retry)
2020-04-16 08:35:05.578 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/transport.py",
line 133, in _send
2020-04-16 08:35:05.578 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=retry)
2020-04-16 08:35:05.578 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 584, in send
2020-04-16 08:35:05.578 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
call_monitor_timeout, retry=retry)
2020-04-16 08:35:05.578 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 573, in _send
2020-04-16 08:35:05.578 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
call_monitor_timeout)
2020-04-16 08:35:05.578 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 459, in wait
2020-04-16 08:35:05.578 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent message =
self.waiters.get(msg_id, timeout=timeout)
2020-04-16 08:35:05.578 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 336, in get
2020-04-16 08:35:05.578 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 'to message
ID %s' % msg_id)
2020-04-16 08:35:05.578 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
MessagingTimeout: Timed out waiting for a reply to message ID
ede4a709807e43dc90df4469ca9d88f3
2020-04-16 08:35:05.578 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2020-04-16 08:35:05.578 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Failed
reporting state!: MessagingTimeout: Timed out waiting for a reply to message ID
ede4a709807e43dc90df4469ca9d88f3
2020-04-16 08:35:05.578 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most
recent call last):
2020-04-16 08:35:05.578 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 328, in _report_state
2020-04-16 08:35:05.578 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent True)
2020-04-16 08:35:05.578 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/rpc.py",
line 97, in report_state
2020-04-16 08:35:05.578 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return
method(context, 'report_state', **kwargs)
2020-04-16 08:35:05.578 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py",
line 179, in call
2020-04-16 08:35:05.578 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
retry=self.retry)
2020-04-16 08:35:05.578 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/transport.py",
line 133, in _send
2020-04-16 08:35:05.578 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=retry)
2020-04-16 08:35:05.578 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 584, in send
2020-04-16 08:35:05.578 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
call_monitor_timeout, retry=retry)
2020-04-16 08:35:05.578 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 573, in _send
2020-04-16 08:35:05.578 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
call_monitor_timeout)
2020-04-16 08:35:05.578 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 459, in wait
2020-04-16 08:35:05.578 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent message =
self.waiters.get(msg_id, timeout=timeout)
2020-04-16 08:35:05.578 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 336, in get
2020-04-16 08:35:05.578 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 'to message
ID %s' % msg_id)
2020-04-16 08:35:05.578 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
MessagingTimeout: Timed out waiting for a reply to message ID
ede4a709807e43dc90df4469ca9d88f3
2020-04-16 08:35:05.578 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2020-04-16 08:35:05.586 21720 WARNING oslo.service.loopingcall [-] Function
'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent._report_state'
run outlasted interval by 30.01 sec
2020-04-16 08:35:44.208 21720 ERROR neutron.common.rpc
[req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Timeout in RPC method
update_device_list. Waiting for 40 seconds before next attempt. If the server
is not down, consider increasing the rpc_response_timeout option as Neutron
server(s) may be overloaded and unable to respond quickly enough.:
MessagingTimeout: Timed out waiting for a reply to message ID
c1bb1fe4c2fe42259dd3726e7ff9e48a
2020-04-16 08:35:44.208 21720 ERROR neutron.common.rpc
[req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Timeout in RPC method
update_device_list. Waiting for 40 seconds before next attempt. If the server
is not down, consider increasing the rpc_response_timeout option as Neutron
server(s) may be overloaded and unable to respond quickly enough.:
MessagingTimeout: Timed out waiting for a reply to message ID
c1bb1fe4c2fe42259dd3726e7ff9e48a
2020-04-16 08:35:44.216 21720 WARNING neutron.common.rpc
[req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Increasing timeout for
update_device_list calls to 120 seconds. Restart the agent to restore it to the
default value.: MessagingTimeout: Timed out waiting for a reply to message ID
c1bb1fe4c2fe42259dd3726e7ff9e48a
2020-04-16 08:35:44.216 21720 WARNING neutron.common.rpc
[req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Increasing timeout for
update_device_list calls to 120 seconds. Restart the agent to restore it to the
default value.: MessagingTimeout: Timed out waiting for a reply to message ID
c1bb1fe4c2fe42259dd3726e7ff9e48a
2020-04-16 08:36:05.413 21720 INFO oslo_messaging._drivers.amqpdriver [-] No
calling threads waiting for msg_id : e9e7407c173a438096eb1fad7e22d9fb
2020-04-16 08:36:05.592 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Failed
reporting state!: MessagingTimeout: Timed out waiting for a reply to message ID
cc669410f7a345649ca8c653f8ccf939
2020-04-16 08:36:05.592 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most
recent call last):
2020-04-16 08:36:05.592 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 328, in _report_state
2020-04-16 08:36:05.592 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent True)
2020-04-16 08:36:05.592 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/rpc.py",
line 97, in report_state
2020-04-16 08:36:05.592 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return
method(context, 'report_state', **kwargs)
2020-04-16 08:36:05.592 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py",
line 179, in call
2020-04-16 08:36:05.592 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
retry=self.retry)
2020-04-16 08:36:05.592 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/transport.py",
line 133, in _send
2020-04-16 08:36:05.592 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=retry)
2020-04-16 08:36:05.592 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 584, in send
2020-04-16 08:36:05.592 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
call_monitor_timeout, retry=retry)
2020-04-16 08:36:05.592 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 573, in _send
2020-04-16 08:36:05.592 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
call_monitor_timeout)
2020-04-16 08:36:05.592 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 459, in wait
2020-04-16 08:36:05.592 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent message =
self.waiters.get(msg_id, timeout=timeout)
2020-04-16 08:36:05.592 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 336, in get
2020-04-16 08:36:05.592 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 'to message
ID %s' % msg_id)
2020-04-16 08:36:05.592 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
MessagingTimeout: Timed out waiting for a reply to message ID
cc669410f7a345649ca8c653f8ccf939
2020-04-16 08:36:05.592 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2020-04-16 08:36:05.592 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Failed
reporting state!: MessagingTimeout: Timed out waiting for a reply to message ID
cc669410f7a345649ca8c653f8ccf939
2020-04-16 08:36:05.592 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most
recent call last):
2020-04-16 08:36:05.592 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 328, in _report_state
2020-04-16 08:36:05.592 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent True)
2020-04-16 08:36:05.592 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/rpc.py",
line 97, in report_state
2020-04-16 08:36:05.592 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return
method(context, 'report_state', **kwargs)
2020-04-16 08:36:05.592 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py",
line 179, in call
2020-04-16 08:36:05.592 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
retry=self.retry)
2020-04-16 08:36:05.592 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/transport.py",
line 133, in _send
2020-04-16 08:36:05.592 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=retry)
2020-04-16 08:36:05.592 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 584, in send
2020-04-16 08:36:05.592 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
call_monitor_timeout, retry=retry)
2020-04-16 08:36:05.592 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 573, in _send
2020-04-16 08:36:05.592 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
call_monitor_timeout)
2020-04-16 08:36:05.592 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 459, in wait
2020-04-16 08:36:05.592 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent message =
self.waiters.get(msg_id, timeout=timeout)
2020-04-16 08:36:05.592 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 336, in get
2020-04-16 08:36:05.592 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 'to message
ID %s' % msg_id)
2020-04-16 08:36:05.592 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
MessagingTimeout: Timed out waiting for a reply to message ID
cc669410f7a345649ca8c653f8ccf939
2020-04-16 08:36:05.592 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2020-04-16 08:36:05.600 21720 WARNING oslo.service.loopingcall [-] Function
'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent._report_state'
run outlasted interval by 30.01 sec
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
[req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Error while processing VIF
ports: MessagingTimeout: Timed out waiting for a reply to message ID
c1bb1fe4c2fe42259dd3726e7ff9e48a
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most
recent call last):
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 2236, in rpc_loop
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info,
provisioning_needed)
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/osprofiler/profiler.py",
line 159, in wrapper
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent result =
f(*args, **kwargs)
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 1802, in process_network_ports
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
failed_devices['added'] |= self._bind_devices(need_binding_devices)
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 938, in _bind_devices
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
self.conf.host, agent_restarted=agent_restarted)
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/rpc.py",
line 170, in update_device_list
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
agent_restarted=agent_restarted)
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/neutron/common/rpc.py",
line 173, in call
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
time.sleep(wait)
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/oslo_utils/excutils.py",
line 220, in __exit__
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
self.force_reraise()
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
six.reraise(self.type_, self.value, self.tb)
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/neutron/common/rpc.py",
line 150, in call
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return
self._original_context.call(ctxt, method, **kwargs)
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py",
line 179, in call
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
retry=self.retry)
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/transport.py",
line 133, in _send
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=retry)
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 584, in send
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
call_monitor_timeout, retry=retry)
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 573, in _send
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
call_monitor_timeout)
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 459, in wait
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent message =
self.waiters.get(msg_id, timeout=timeout)
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 336, in get
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 'to message
ID %s' % msg_id)
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
MessagingTimeout: Timed out waiting for a reply to message ID
c1bb1fe4c2fe42259dd3726e7ff9e48a
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
[req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Error while processing VIF
ports: MessagingTimeout: Timed out waiting for a reply to message ID
c1bb1fe4c2fe42259dd3726e7ff9e48a
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most
recent call last):
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 2236, in rpc_loop
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info,
provisioning_needed)
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/osprofiler/profiler.py",
line 159, in wrapper
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent result =
f(*args, **kwargs)
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 1802, in process_network_ports
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
failed_devices['added'] |= self._bind_devices(need_binding_devices)
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 938, in _bind_devices
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
self.conf.host, agent_restarted=agent_restarted)
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/rpc.py",
line 170, in update_device_list
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
agent_restarted=agent_restarted)
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/neutron/common/rpc.py",
line 173, in call
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
time.sleep(wait)
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/oslo_utils/excutils.py",
line 220, in __exit__
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
self.force_reraise()
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
six.reraise(self.type_, self.value, self.tb)
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/neutron/common/rpc.py",
line 150, in call
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return
self._original_context.call(ctxt, method, **kwargs)
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py",
line 179, in call
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
retry=self.retry)
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/transport.py",
line 133, in _send
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=retry)
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 584, in send
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
call_monitor_timeout, retry=retry)
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 573, in _send
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
call_monitor_timeout)
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 459, in wait
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent message =
self.waiters.get(msg_id, timeout=timeout)
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 336, in get
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 'to message
ID %s' % msg_id)
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
MessagingTimeout: Timed out waiting for a reply to message ID
c1bb1fe4c2fe42259dd3726e7ff9e48a
2020-04-16 08:36:24.558 21720 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2020-04-16 08:36:24.568 21720 DEBUG
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
[req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Agent rpc_loop -
iteration:0 completed. Processed ports statistics: {'regular': {'updated': 0,
'added': 10, 'removed': 0}}. Elapsed:137.142 loop_count_and_wait
/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:1944
2020-04-16 08:36:24.568 21720 DEBUG
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
[req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Agent rpc_loop -
iteration:0 completed. Processed ports statistics: {'regular': {'updated': 0,
'added': 10, 'removed': 0}}. Elapsed:137.142 loop_count_and_wait
/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:1944
2020-04-16 08:36:24.575 21720 DEBUG
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
[req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Loop iteration exceeded
interval (2 vs. 137.142431974)! loop_count_and_wait
/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:1951
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1873265/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
