Public bug reported:
Hi All,
I'm using Ocata version. I deployed several compute nodes and two
active/backup controller nodes with running neutron-server and rabbitmq.
after I switched the role of the controller nodes. the dhcp-agent
disabled its report state. here is dhcp-agent log:
2018-01-23 21:07:59.542 2903 ERROR oslo.messaging._drivers.impl_rabbit
[req-d3517c03-3118-43db-ae21-dc28e29ae79b - - - - -]
[f4c046e5-9ddf-4f67-b4f2-3064e47fc9d8] AMQP server on 100.7.38.246:5672 is
unreachable: timed out. Trying again in 2 seconds. Client port: None: timeout:
timed out
2018-01-23 21:08:01.489 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:08:01.554 2903 ERROR oslo.messaging._drivers.impl_rabbit
[req-d3517c03-3118-43db-ae21-dc28e29ae79b - - - - -]
[f4c046e5-9ddf-4f67-b4f2-3064e47fc9d8] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:08:02.481 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:08:05.503 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:08:05.568 2903 ERROR oslo.messaging._drivers.impl_rabbit
[req-d3517c03-3118-43db-ae21-dc28e29ae79b - - - - -]
[f4c046e5-9ddf-4f67-b4f2-3064e47fc9d8] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:08:06.494 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:08:11.547 2903 INFO oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] Reconnected to AMQP server on
100.7.38.246:5672 via [amqp] client with port 47254.
2018-01-23 21:08:11.599 2903 INFO oslo.messaging._drivers.impl_rabbit
[req-d3517c03-3118-43db-ae21-dc28e29ae79b - - - - -]
[f4c046e5-9ddf-4f67-b4f2-3064e47fc9d8] Reconnected to AMQP server on
100.7.38.246:5672 via [amqp] client with port 47256.
2018-01-23 21:08:12.541 2903 INFO oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] Reconnected to AMQP server on
100.7.38.246:5672 via [amqp] client with port 47258.
2018-01-23 21:08:32.154 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" acquired by
"neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-01-23 21:08:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" released by
"neutron.agent.linux.external_process._check_child_processes" :: held 0.001s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2018-01-23 21:09:32.154 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" acquired by
"neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-01-23 21:09:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" released by
"neutron.agent.linux.external_process._check_child_processes" :: held 0.001s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2018-01-23 21:10:32.154 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" acquired by
"neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-01-23 21:10:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" released by
"neutron.agent.linux.external_process._check_child_processes" :: held 0.001s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2018-01-23 21:10:53.561 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure
with reason 'shutdown'. Trying again in 1 seconds. Client port: None:
ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection
closure with reason 'shutdown'
2018-01-23 21:10:53.570 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure
with reason 'shutdown'. Trying again in 1 seconds. Client port: None:
ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection
closure with reason 'shutdown'
2018-01-23 21:10:54.584 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:10:54.586 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:10:56.608 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:10:56.609 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client
port: None: RecoverableConnectionError: <AMQPError: unknown error>
2018-01-23 21:10:56.623 2903 WARNING oslo.messaging._drivers.impl_rabbit [-]
Unexpected error during heartbeart thread processing, retrying...:
ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection
closure with reason 'shutdown'
2018-01-23 21:10:57.621 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:10:59.634 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:11:00.621 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:11:08.537 2903 WARNING oslo.messaging._drivers.impl_rabbit [-]
Unexpected error during heartbeart thread processing, retrying...: error:
[Errno 32] Broken pipe
2018-01-23 21:11:08.648 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client
port: None: RecoverableConnectionError: <AMQPError: unknown error>
2018-01-23 21:11:11.635 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: timed out. Trying again in 8 seconds. Client port: None: timeout:
timed out
2018-01-23 21:11:14.659 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: timed out. Trying again in 2 seconds. Client port: None: timeout:
timed out
2018-01-23 21:11:16.635 2903 WARNING oslo.messaging._drivers.impl_rabbit [-]
Unexpected error during heartbeart thread processing, retrying...: timeout:
timed out
2018-01-23 21:11:21.673 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: timed out. Trying again in 4 seconds. Client port: None: timeout:
timed out
2018-01-23 21:11:22.709 2903 INFO oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] Reconnected to AMQP server on
100.7.38.246:5672 via [amqp] client with port 47434.
2018-01-23 21:11:23.524 2903 ERROR oslo.messaging._drivers.impl_rabbit
[req-968a5271-5b02-43a9-93a8-8e1565c2e9b3 - - - - -]
[f4c046e5-9ddf-4f67-b4f2-3064e47fc9d8] AMQP server on 100.7.38.246:5672 is
unreachable: basic_publish: connection closed. Trying again in 1 seconds.
Client port: None: RecoverableConnectionError: basic_publish: connection closed
2018-01-23 21:11:24.558 2903 INFO oslo.messaging._drivers.impl_rabbit
[req-968a5271-5b02-43a9-93a8-8e1565c2e9b3 - - - - -]
[f4c046e5-9ddf-4f67-b4f2-3064e47fc9d8] Reconnected to AMQP server on
100.7.38.246:5672 via [amqp] client with port 47458.
2018-01-23 21:11:25.714 2903 INFO oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] Reconnected to AMQP server on
100.7.38.246:5672 via [amqp] client with port 47460.
2018-01-23 21:11:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" acquired by
"neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-01-23 21:11:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" released by
"neutron.agent.linux.external_process._check_child_processes" :: held 0.001s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent
[req-968a5271-5b02-43a9-93a8-8e1565c2e9b3 - - - - -] Failed reporting state!:
MessagingTimeout: Timed out waiting for a reply to message ID
e3054e3b13b244af895ee8b2335ddc85
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent Traceback (most
recent call last):
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python2.7/site-packages/neutron/agent/dhcp/agent.py", line 733, in
_report_state
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent ctx,
self.agent_state, True)
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 87, in
report_state
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent return
method(context, 'report_state', **kwargs)
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in
call
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent
retry=self.retry)
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 123, in
_send
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent
timeout=timeout, retry=retry)
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line
578, in send
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent retry=retry)
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line
567, in _send
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent result =
self._waiter.wait(msg_id, timeout)
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line
459, in wait
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent message =
self.waiters.get(msg_id, timeout=timeout)
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line
347, in get
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent 'to message ID
%s' % msg_id)
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent MessagingTimeout:
Timed out waiting for a reply to message ID e3054e3b13b244af895ee8b2335ddc85
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent
2018-01-23 21:12:24.565 2903 WARNING oslo.service.loopingcall
[req-968a5271-5b02-43a9-93a8-8e1565c2e9b3 - - - - -] Function
'neutron.agent.dhcp.agent.DhcpAgentWithStateReport._report_state' run outlasted
interval by 36.04 sec
2018-01-23 21:12:24.656 2903 INFO neutron.agent.dhcp.agent
[req-6ae984dc-fcf6-45a7-8a00-800d9e6a0eb1 - - - - -] Agent has just been
revived. Scheduling full sync
2018-01-23 21:12:29.132 2903 DEBUG neutron.agent.dhcp.agent
[req-58690281-b59d-4a71-a190-d0a11dde6e0f - - - - -] resync (*): ['Agent has
just been revived'] _periodic_resync_helper
/usr/lib/python2.7/site-packages/neutron/agent/dhcp/agent.py:261
2018-01-23 21:12:29.133 2903 INFO neutron.agent.dhcp.agent
[req-58690281-b59d-4a71-a190-d0a11dde6e0f - - - - -] Synchronizing state
2018-01-23 21:12:29.518 2903 INFO neutron.agent.dhcp.agent
[req-a4f75792-796e-474a-9bac-5d13fbb31197 - - - - -] All active networks have
been fetched through RPC.
2018-01-23 21:12:29.519 2903 INFO neutron.agent.dhcp.agent
[req-a4f75792-796e-474a-9bac-5d13fbb31197 - - - - -] Synchronizing state
complete
2018-01-23 21:12:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" acquired by
"neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-01-23 21:12:32.156 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" released by
"neutron.agent.linux.external_process._check_child_processes" :: held 0.001s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2018-01-23 21:13:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" acquired by
"neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-01-23 21:13:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" released by
"neutron.agent.linux.external_process._check_child_processes" :: held 0.001s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2018-01-23 21:13:59.991 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure
with reason 'shutdown'. Trying again in 1 seconds. Client port: None:
ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection
closure with reason 'shutdown'
2018-01-23 21:13:59.996 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure
with reason 'shutdown'. Trying again in 1 seconds. Client port: None:
ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection
closure with reason 'shutdown'
2018-01-23 21:14:01.015 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:14:01.016 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:14:01.653 2903 WARNING oslo.messaging._drivers.impl_rabbit [-]
Unexpected error during heartbeart thread processing, retrying...:
ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection
closure with reason 'shutdown'
2018-01-23 21:14:03.029 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client
port: None: RecoverableConnectionError: <AMQPError: unknown error>
2018-01-23 21:14:03.041 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:14:04.041 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:14:06.055 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:14:07.055 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:14:08.586 2903 WARNING oslo.messaging._drivers.impl_rabbit [-]
Unexpected error during heartbeart thread processing, retrying...: error:
[Errno 32] Broken pipe
2018-01-23 21:14:15.069 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client
port: None: RecoverableConnectionError: <AMQPError: unknown error>
2018-01-23 21:14:18.068 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: timed out. Trying again in 8 seconds. Client port: None: timeout:
timed out
2018-01-23 21:14:21.083 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: timed out. Trying again in 2 seconds. Client port: None: timeout:
timed out
2018-01-23 21:14:21.666 2903 WARNING oslo.messaging._drivers.impl_rabbit [-]
Unexpected error during heartbeart thread processing, retrying...: timeout:
timed out
2018-01-23 21:14:28.096 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: timed out. Trying again in 4 seconds. Client port: None: timeout:
timed out
2018-01-23 21:14:28.597 2903 WARNING oslo.messaging._drivers.impl_rabbit [-]
Unexpected error during heartbeart thread processing, retrying...: timeout:
timed out
2018-01-23 21:14:29.086 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client
port: None: RecoverableConnectionError: <AMQPError: unknown error>
2018-01-23 21:14:29.577 2903 ERROR oslo.messaging._drivers.impl_rabbit
[req-1bc1ad73-fc9d-4911-b170-f94c2f62514c - - - - -] Failed to publish message
to topic 'neutron': 'NoneType' object has no attribute 'itervalues'
2018-01-23 21:14:29.578 2903 WARNING neutron.agent.dhcp.agent
[req-1bc1ad73-fc9d-4911-b170-f94c2f62514c - - - - -] Neutron server does not
support state report. State report for this agent will be disabled.
2018-01-23 21:14:30.099 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:14:32.122 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client
port: None: RecoverableConnectionError: <AMQPError: unknown error>
2018-01-23 21:14:32.123 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:14:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" acquired by
"neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-01-23 21:14:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" released by
"neutron.agent.linux.external_process._check_child_processes" :: held 0.000s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2018-01-23 21:14:33.134 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:14:35.175 2903 INFO oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] Reconnected to AMQP server on
100.7.38.246:5672 via [amqp] client with port 47694.
2018-01-23 21:14:36.191 2903 INFO oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] Reconnected to AMQP server on
100.7.38.246:5672 via [amqp] client with port 47696.
2018-01-23 21:15:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" acquired by
"neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-01-23 21:15:32.156 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" released by
"neutron.agent.linux.external_process._check_child_processes" :: held 0.001s
inner /usr/lib/pytho
** Affects: neutron
Importance: Undecided
Status: New
** Description changed:
Hi All,
I'm using Ocata version. I deployed several compute nodes and two
- active/backup controller node with running neutron-server and rabbitmq.
+ active/backup controller nodes with running neutron-server and rabbitmq.
after I switched the role of the controller nodes. the dhcp-agent
disabled its report state. here is dhcp-agent log:
2018-01-23 21:07:59.542 2903 ERROR oslo.messaging._drivers.impl_rabbit
[req-d3517c03-3118-43db-ae21-dc28e29ae79b - - - - -]
[f4c046e5-9ddf-4f67-b4f2-3064e47fc9d8] AMQP server on 100.7.38.246:5672 is
unreachable: timed out. Trying again in 2 seconds. Client port: None: timeout:
timed out
2018-01-23 21:08:01.489 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:08:01.554 2903 ERROR oslo.messaging._drivers.impl_rabbit
[req-d3517c03-3118-43db-ae21-dc28e29ae79b - - - - -]
[f4c046e5-9ddf-4f67-b4f2-3064e47fc9d8] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:08:02.481 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:08:05.503 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:08:05.568 2903 ERROR oslo.messaging._drivers.impl_rabbit
[req-d3517c03-3118-43db-ae21-dc28e29ae79b - - - - -]
[f4c046e5-9ddf-4f67-b4f2-3064e47fc9d8] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:08:06.494 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:08:11.547 2903 INFO oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] Reconnected to AMQP server on
100.7.38.246:5672 via [amqp] client with port 47254.
2018-01-23 21:08:11.599 2903 INFO oslo.messaging._drivers.impl_rabbit
[req-d3517c03-3118-43db-ae21-dc28e29ae79b - - - - -]
[f4c046e5-9ddf-4f67-b4f2-3064e47fc9d8] Reconnected to AMQP server on
100.7.38.246:5672 via [amqp] client with port 47256.
2018-01-23 21:08:12.541 2903 INFO oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] Reconnected to AMQP server on
100.7.38.246:5672 via [amqp] client with port 47258.
2018-01-23 21:08:32.154 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" acquired by
"neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-01-23 21:08:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" released by
"neutron.agent.linux.external_process._check_child_processes" :: held 0.001s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2018-01-23 21:09:32.154 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" acquired by
"neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-01-23 21:09:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" released by
"neutron.agent.linux.external_process._check_child_processes" :: held 0.001s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2018-01-23 21:10:32.154 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" acquired by
"neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-01-23 21:10:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" released by
"neutron.agent.linux.external_process._check_child_processes" :: held 0.001s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2018-01-23 21:10:53.561 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure
with reason 'shutdown'. Trying again in 1 seconds. Client port: None:
ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection
closure with reason 'shutdown'
2018-01-23 21:10:53.570 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure
with reason 'shutdown'. Trying again in 1 seconds. Client port: None:
ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection
closure with reason 'shutdown'
2018-01-23 21:10:54.584 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:10:54.586 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:10:56.608 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:10:56.609 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client
port: None: RecoverableConnectionError: <AMQPError: unknown error>
2018-01-23 21:10:56.623 2903 WARNING oslo.messaging._drivers.impl_rabbit [-]
Unexpected error during heartbeart thread processing, retrying...:
ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection
closure with reason 'shutdown'
2018-01-23 21:10:57.621 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:10:59.634 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:11:00.621 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:11:08.537 2903 WARNING oslo.messaging._drivers.impl_rabbit [-]
Unexpected error during heartbeart thread processing, retrying...: error:
[Errno 32] Broken pipe
2018-01-23 21:11:08.648 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client
port: None: RecoverableConnectionError: <AMQPError: unknown error>
2018-01-23 21:11:11.635 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: timed out. Trying again in 8 seconds. Client port: None: timeout:
timed out
2018-01-23 21:11:14.659 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: timed out. Trying again in 2 seconds. Client port: None: timeout:
timed out
2018-01-23 21:11:16.635 2903 WARNING oslo.messaging._drivers.impl_rabbit [-]
Unexpected error during heartbeart thread processing, retrying...: timeout:
timed out
2018-01-23 21:11:21.673 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: timed out. Trying again in 4 seconds. Client port: None: timeout:
timed out
2018-01-23 21:11:22.709 2903 INFO oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] Reconnected to AMQP server on
100.7.38.246:5672 via [amqp] client with port 47434.
2018-01-23 21:11:23.524 2903 ERROR oslo.messaging._drivers.impl_rabbit
[req-968a5271-5b02-43a9-93a8-8e1565c2e9b3 - - - - -]
[f4c046e5-9ddf-4f67-b4f2-3064e47fc9d8] AMQP server on 100.7.38.246:5672 is
unreachable: basic_publish: connection closed. Trying again in 1 seconds.
Client port: None: RecoverableConnectionError: basic_publish: connection closed
2018-01-23 21:11:24.558 2903 INFO oslo.messaging._drivers.impl_rabbit
[req-968a5271-5b02-43a9-93a8-8e1565c2e9b3 - - - - -]
[f4c046e5-9ddf-4f67-b4f2-3064e47fc9d8] Reconnected to AMQP server on
100.7.38.246:5672 via [amqp] client with port 47458.
2018-01-23 21:11:25.714 2903 INFO oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] Reconnected to AMQP server on
100.7.38.246:5672 via [amqp] client with port 47460.
2018-01-23 21:11:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" acquired by
"neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-01-23 21:11:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" released by
"neutron.agent.linux.external_process._check_child_processes" :: held 0.001s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent
[req-968a5271-5b02-43a9-93a8-8e1565c2e9b3 - - - - -] Failed reporting state!:
MessagingTimeout: Timed out waiting for a reply to message ID
e3054e3b13b244af895ee8b2335ddc85
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent Traceback (most
recent call last):
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python2.7/site-packages/neutron/agent/dhcp/agent.py", line 733, in
_report_state
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent ctx,
self.agent_state, True)
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 87, in
report_state
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent return
method(context, 'report_state', **kwargs)
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in
call
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent
retry=self.retry)
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 123, in
_send
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent
timeout=timeout, retry=retry)
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line
578, in send
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent retry=retry)
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line
567, in _send
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent result =
self._waiter.wait(msg_id, timeout)
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line
459, in wait
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent message =
self.waiters.get(msg_id, timeout=timeout)
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line
347, in get
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent 'to message
ID %s' % msg_id)
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent MessagingTimeout:
Timed out waiting for a reply to message ID e3054e3b13b244af895ee8b2335ddc85
- 2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent
+ 2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent
2018-01-23 21:12:24.565 2903 WARNING oslo.service.loopingcall
[req-968a5271-5b02-43a9-93a8-8e1565c2e9b3 - - - - -] Function
'neutron.agent.dhcp.agent.DhcpAgentWithStateReport._report_state' run outlasted
interval by 36.04 sec
2018-01-23 21:12:24.656 2903 INFO neutron.agent.dhcp.agent
[req-6ae984dc-fcf6-45a7-8a00-800d9e6a0eb1 - - - - -] Agent has just been
revived. Scheduling full sync
2018-01-23 21:12:29.132 2903 DEBUG neutron.agent.dhcp.agent
[req-58690281-b59d-4a71-a190-d0a11dde6e0f - - - - -] resync (*): ['Agent has
just been revived'] _periodic_resync_helper
/usr/lib/python2.7/site-packages/neutron/agent/dhcp/agent.py:261
2018-01-23 21:12:29.133 2903 INFO neutron.agent.dhcp.agent
[req-58690281-b59d-4a71-a190-d0a11dde6e0f - - - - -] Synchronizing state
2018-01-23 21:12:29.518 2903 INFO neutron.agent.dhcp.agent
[req-a4f75792-796e-474a-9bac-5d13fbb31197 - - - - -] All active networks have
been fetched through RPC.
2018-01-23 21:12:29.519 2903 INFO neutron.agent.dhcp.agent
[req-a4f75792-796e-474a-9bac-5d13fbb31197 - - - - -] Synchronizing state
complete
2018-01-23 21:12:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" acquired by
"neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-01-23 21:12:32.156 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" released by
"neutron.agent.linux.external_process._check_child_processes" :: held 0.001s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2018-01-23 21:13:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" acquired by
"neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-01-23 21:13:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" released by
"neutron.agent.linux.external_process._check_child_processes" :: held 0.001s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2018-01-23 21:13:59.991 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure
with reason 'shutdown'. Trying again in 1 seconds. Client port: None:
ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection
closure with reason 'shutdown'
2018-01-23 21:13:59.996 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure
with reason 'shutdown'. Trying again in 1 seconds. Client port: None:
ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection
closure with reason 'shutdown'
2018-01-23 21:14:01.015 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:14:01.016 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:14:01.653 2903 WARNING oslo.messaging._drivers.impl_rabbit [-]
Unexpected error during heartbeart thread processing, retrying...:
ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection
closure with reason 'shutdown'
2018-01-23 21:14:03.029 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client
port: None: RecoverableConnectionError: <AMQPError: unknown error>
2018-01-23 21:14:03.041 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:14:04.041 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:14:06.055 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:14:07.055 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:14:08.586 2903 WARNING oslo.messaging._drivers.impl_rabbit [-]
Unexpected error during heartbeart thread processing, retrying...: error:
[Errno 32] Broken pipe
2018-01-23 21:14:15.069 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client
port: None: RecoverableConnectionError: <AMQPError: unknown error>
2018-01-23 21:14:18.068 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: timed out. Trying again in 8 seconds. Client port: None: timeout:
timed out
2018-01-23 21:14:21.083 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: timed out. Trying again in 2 seconds. Client port: None: timeout:
timed out
2018-01-23 21:14:21.666 2903 WARNING oslo.messaging._drivers.impl_rabbit [-]
Unexpected error during heartbeart thread processing, retrying...: timeout:
timed out
2018-01-23 21:14:28.096 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: timed out. Trying again in 4 seconds. Client port: None: timeout:
timed out
2018-01-23 21:14:28.597 2903 WARNING oslo.messaging._drivers.impl_rabbit [-]
Unexpected error during heartbeart thread processing, retrying...: timeout:
timed out
2018-01-23 21:14:29.086 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client
port: None: RecoverableConnectionError: <AMQPError: unknown error>
2018-01-23 21:14:29.577 2903 ERROR oslo.messaging._drivers.impl_rabbit
[req-1bc1ad73-fc9d-4911-b170-f94c2f62514c - - - - -] Failed to publish message
to topic 'neutron': 'NoneType' object has no attribute 'itervalues'
2018-01-23 21:14:29.578 2903 WARNING neutron.agent.dhcp.agent
[req-1bc1ad73-fc9d-4911-b170-f94c2f62514c - - - - -] Neutron server does not
support state report. State report for this agent will be disabled.
2018-01-23 21:14:30.099 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:14:32.122 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client
port: None: RecoverableConnectionError: <AMQPError: unknown error>
2018-01-23 21:14:32.123 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:14:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" acquired by
"neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-01-23 21:14:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" released by
"neutron.agent.linux.external_process._check_child_processes" :: held 0.000s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2018-01-23 21:14:33.134 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:14:35.175 2903 INFO oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] Reconnected to AMQP server on
100.7.38.246:5672 via [amqp] client with port 47694.
2018-01-23 21:14:36.191 2903 INFO oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] Reconnected to AMQP server on
100.7.38.246:5672 via [amqp] client with port 47696.
2018-01-23 21:15:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" acquired by
"neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-01-23 21:15:32.156 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" released by
"neutron.agent.linux.external_process._check_child_processes" :: held 0.001s
inner /usr/lib/pytho
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1745262
Title:
State report fo dhcp agent was disabled after active/backup switch
serveral times
Status in neutron:
New
Bug description:
Hi All,
I'm using Ocata version. I deployed several compute nodes and two
active/backup controller nodes with running neutron-server and
rabbitmq. after I switched the role of the controller nodes. the dhcp-
agent disabled its report state. here is dhcp-agent log:
2018-01-23 21:07:59.542 2903 ERROR oslo.messaging._drivers.impl_rabbit
[req-d3517c03-3118-43db-ae21-dc28e29ae79b - - - - -]
[f4c046e5-9ddf-4f67-b4f2-3064e47fc9d8] AMQP server on 100.7.38.246:5672 is
unreachable: timed out. Trying again in 2 seconds. Client port: None: timeout:
timed out
2018-01-23 21:08:01.489 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:08:01.554 2903 ERROR oslo.messaging._drivers.impl_rabbit
[req-d3517c03-3118-43db-ae21-dc28e29ae79b - - - - -]
[f4c046e5-9ddf-4f67-b4f2-3064e47fc9d8] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:08:02.481 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:08:05.503 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:08:05.568 2903 ERROR oslo.messaging._drivers.impl_rabbit
[req-d3517c03-3118-43db-ae21-dc28e29ae79b - - - - -]
[f4c046e5-9ddf-4f67-b4f2-3064e47fc9d8] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:08:06.494 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:08:11.547 2903 INFO oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] Reconnected to AMQP server on
100.7.38.246:5672 via [amqp] client with port 47254.
2018-01-23 21:08:11.599 2903 INFO oslo.messaging._drivers.impl_rabbit
[req-d3517c03-3118-43db-ae21-dc28e29ae79b - - - - -]
[f4c046e5-9ddf-4f67-b4f2-3064e47fc9d8] Reconnected to AMQP server on
100.7.38.246:5672 via [amqp] client with port 47256.
2018-01-23 21:08:12.541 2903 INFO oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] Reconnected to AMQP server on
100.7.38.246:5672 via [amqp] client with port 47258.
2018-01-23 21:08:32.154 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" acquired by
"neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-01-23 21:08:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" released by
"neutron.agent.linux.external_process._check_child_processes" :: held 0.001s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2018-01-23 21:09:32.154 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" acquired by
"neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-01-23 21:09:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" released by
"neutron.agent.linux.external_process._check_child_processes" :: held 0.001s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2018-01-23 21:10:32.154 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" acquired by
"neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-01-23 21:10:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" released by
"neutron.agent.linux.external_process._check_child_processes" :: held 0.001s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2018-01-23 21:10:53.561 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure
with reason 'shutdown'. Trying again in 1 seconds. Client port: None:
ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection
closure with reason 'shutdown'
2018-01-23 21:10:53.570 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure
with reason 'shutdown'. Trying again in 1 seconds. Client port: None:
ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection
closure with reason 'shutdown'
2018-01-23 21:10:54.584 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:10:54.586 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:10:56.608 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:10:56.609 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client
port: None: RecoverableConnectionError: <AMQPError: unknown error>
2018-01-23 21:10:56.623 2903 WARNING oslo.messaging._drivers.impl_rabbit [-]
Unexpected error during heartbeart thread processing, retrying...:
ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection
closure with reason 'shutdown'
2018-01-23 21:10:57.621 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:10:59.634 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:11:00.621 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:11:08.537 2903 WARNING oslo.messaging._drivers.impl_rabbit [-]
Unexpected error during heartbeart thread processing, retrying...: error:
[Errno 32] Broken pipe
2018-01-23 21:11:08.648 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client
port: None: RecoverableConnectionError: <AMQPError: unknown error>
2018-01-23 21:11:11.635 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: timed out. Trying again in 8 seconds. Client port: None: timeout:
timed out
2018-01-23 21:11:14.659 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: timed out. Trying again in 2 seconds. Client port: None: timeout:
timed out
2018-01-23 21:11:16.635 2903 WARNING oslo.messaging._drivers.impl_rabbit [-]
Unexpected error during heartbeart thread processing, retrying...: timeout:
timed out
2018-01-23 21:11:21.673 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: timed out. Trying again in 4 seconds. Client port: None: timeout:
timed out
2018-01-23 21:11:22.709 2903 INFO oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] Reconnected to AMQP server on
100.7.38.246:5672 via [amqp] client with port 47434.
2018-01-23 21:11:23.524 2903 ERROR oslo.messaging._drivers.impl_rabbit
[req-968a5271-5b02-43a9-93a8-8e1565c2e9b3 - - - - -]
[f4c046e5-9ddf-4f67-b4f2-3064e47fc9d8] AMQP server on 100.7.38.246:5672 is
unreachable: basic_publish: connection closed. Trying again in 1 seconds.
Client port: None: RecoverableConnectionError: basic_publish: connection closed
2018-01-23 21:11:24.558 2903 INFO oslo.messaging._drivers.impl_rabbit
[req-968a5271-5b02-43a9-93a8-8e1565c2e9b3 - - - - -]
[f4c046e5-9ddf-4f67-b4f2-3064e47fc9d8] Reconnected to AMQP server on
100.7.38.246:5672 via [amqp] client with port 47458.
2018-01-23 21:11:25.714 2903 INFO oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] Reconnected to AMQP server on
100.7.38.246:5672 via [amqp] client with port 47460.
2018-01-23 21:11:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" acquired by
"neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-01-23 21:11:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" released by
"neutron.agent.linux.external_process._check_child_processes" :: held 0.001s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent
[req-968a5271-5b02-43a9-93a8-8e1565c2e9b3 - - - - -] Failed reporting state!:
MessagingTimeout: Timed out waiting for a reply to message ID
e3054e3b13b244af895ee8b2335ddc85
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent Traceback (most
recent call last):
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python2.7/site-packages/neutron/agent/dhcp/agent.py", line 733, in
_report_state
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent ctx,
self.agent_state, True)
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 87, in
report_state
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent return
method(context, 'report_state', **kwargs)
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in
call
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent
retry=self.retry)
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 123, in
_send
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent
timeout=timeout, retry=retry)
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line
578, in send
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent retry=retry)
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line
567, in _send
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent result =
self._waiter.wait(msg_id, timeout)
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line
459, in wait
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent message =
self.waiters.get(msg_id, timeout=timeout)
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent File
"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line
347, in get
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent 'to message
ID %s' % msg_id)
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent MessagingTimeout:
Timed out waiting for a reply to message ID e3054e3b13b244af895ee8b2335ddc85
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent
2018-01-23 21:12:24.565 2903 WARNING oslo.service.loopingcall
[req-968a5271-5b02-43a9-93a8-8e1565c2e9b3 - - - - -] Function
'neutron.agent.dhcp.agent.DhcpAgentWithStateReport._report_state' run outlasted
interval by 36.04 sec
2018-01-23 21:12:24.656 2903 INFO neutron.agent.dhcp.agent
[req-6ae984dc-fcf6-45a7-8a00-800d9e6a0eb1 - - - - -] Agent has just been
revived. Scheduling full sync
2018-01-23 21:12:29.132 2903 DEBUG neutron.agent.dhcp.agent
[req-58690281-b59d-4a71-a190-d0a11dde6e0f - - - - -] resync (*): ['Agent has
just been revived'] _periodic_resync_helper
/usr/lib/python2.7/site-packages/neutron/agent/dhcp/agent.py:261
2018-01-23 21:12:29.133 2903 INFO neutron.agent.dhcp.agent
[req-58690281-b59d-4a71-a190-d0a11dde6e0f - - - - -] Synchronizing state
2018-01-23 21:12:29.518 2903 INFO neutron.agent.dhcp.agent
[req-a4f75792-796e-474a-9bac-5d13fbb31197 - - - - -] All active networks have
been fetched through RPC.
2018-01-23 21:12:29.519 2903 INFO neutron.agent.dhcp.agent
[req-a4f75792-796e-474a-9bac-5d13fbb31197 - - - - -] Synchronizing state
complete
2018-01-23 21:12:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" acquired by
"neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-01-23 21:12:32.156 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" released by
"neutron.agent.linux.external_process._check_child_processes" :: held 0.001s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2018-01-23 21:13:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" acquired by
"neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-01-23 21:13:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" released by
"neutron.agent.linux.external_process._check_child_processes" :: held 0.001s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2018-01-23 21:13:59.991 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure
with reason 'shutdown'. Trying again in 1 seconds. Client port: None:
ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection
closure with reason 'shutdown'
2018-01-23 21:13:59.996 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure
with reason 'shutdown'. Trying again in 1 seconds. Client port: None:
ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection
closure with reason 'shutdown'
2018-01-23 21:14:01.015 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:14:01.016 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:14:01.653 2903 WARNING oslo.messaging._drivers.impl_rabbit [-]
Unexpected error during heartbeart thread processing, retrying...:
ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection
closure with reason 'shutdown'
2018-01-23 21:14:03.029 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client
port: None: RecoverableConnectionError: <AMQPError: unknown error>
2018-01-23 21:14:03.041 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:14:04.041 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:14:06.055 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:14:07.055 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:14:08.586 2903 WARNING oslo.messaging._drivers.impl_rabbit [-]
Unexpected error during heartbeart thread processing, retrying...: error:
[Errno 32] Broken pipe
2018-01-23 21:14:15.069 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client
port: None: RecoverableConnectionError: <AMQPError: unknown error>
2018-01-23 21:14:18.068 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: timed out. Trying again in 8 seconds. Client port: None: timeout:
timed out
2018-01-23 21:14:21.083 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: timed out. Trying again in 2 seconds. Client port: None: timeout:
timed out
2018-01-23 21:14:21.666 2903 WARNING oslo.messaging._drivers.impl_rabbit [-]
Unexpected error during heartbeart thread processing, retrying...: timeout:
timed out
2018-01-23 21:14:28.096 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: timed out. Trying again in 4 seconds. Client port: None: timeout:
timed out
2018-01-23 21:14:28.597 2903 WARNING oslo.messaging._drivers.impl_rabbit [-]
Unexpected error during heartbeart thread processing, retrying...: timeout:
timed out
2018-01-23 21:14:29.086 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client
port: None: RecoverableConnectionError: <AMQPError: unknown error>
2018-01-23 21:14:29.577 2903 ERROR oslo.messaging._drivers.impl_rabbit
[req-1bc1ad73-fc9d-4911-b170-f94c2f62514c - - - - -] Failed to publish message
to topic 'neutron': 'NoneType' object has no attribute 'itervalues'
2018-01-23 21:14:29.578 2903 WARNING neutron.agent.dhcp.agent
[req-1bc1ad73-fc9d-4911-b170-f94c2f62514c - - - - -] Neutron server does not
support state report. State report for this agent will be disabled.
2018-01-23 21:14:30.099 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:14:32.122 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client
port: None: RecoverableConnectionError: <AMQPError: unknown error>
2018-01-23 21:14:32.123 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:14:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" acquired by
"neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-01-23 21:14:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" released by
"neutron.agent.linux.external_process._check_child_processes" :: held 0.000s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2018-01-23 21:14:33.134 2903 ERROR oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port:
None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:14:35.175 2903 INFO oslo.messaging._drivers.impl_rabbit [-]
[5e001526-90fb-41b4-a5fd-eef0ac96ee00] Reconnected to AMQP server on
100.7.38.246:5672 via [amqp] client with port 47694.
2018-01-23 21:14:36.191 2903 INFO oslo.messaging._drivers.impl_rabbit [-]
[04989bd8-915f-4b83-89e2-e267fe118ce0] Reconnected to AMQP server on
100.7.38.246:5672 via [amqp] client with port 47696.
2018-01-23 21:15:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" acquired by
"neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-01-23 21:15:32.156 2903 DEBUG oslo_concurrency.lockutils [-] Lock
"_check_child_processes" released by
"neutron.agent.linux.external_process._check_child_processes" :: held 0.001s
inner /usr/lib/pytho
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1745262/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
