Kris, You’re right this system work well before. I restarted ,this time, rabbitmq-server, nova-api, nova-conductor ,nova-console, nova-novncproxy and nova-scheduler together in a single line on Controller node. Now it’s back normal. so for lessen learned, if restarting singe service not resolve problem then think about restarting them all if related together.
Thank you Kris. Evan. > On Nov 25, 2015, at 12:12 AM, Kris G. Lindgren <[email protected]> wrote: > > Evan, > > I would recommend restarting anything that takes to rabbitmq. This is a case > of your rpc workers thinking that they are connected to rabbitmq but they > really aren't. I assume that this setup use to work at one time and now > doesn’t? > > ___________________________________________________________________ > Kris Lindgren > Senior Linux Systems Engineer > GoDaddy > > From: Evan <[email protected] <mailto:[email protected]>> > Date: Tuesday, November 24, 2015 at 5:02 AM > To: OpenStack-operators <[email protected] > <mailto:[email protected]>> > Subject: [Openstack-operators] Can't boot image getting nova.api.openstack > MessagingTimeout: Timed out waiting for a reply to message ID msg_id > > hi, > > I keep getting such error(as subject) when trying to boot an image for > instance, no matter from webpage dashboard or cli. firstly it show the status > of scheduling and after a while it's Error. I checked status of all service > ,nothing gets failed. Below are the logs for nova-api and nova-conductor. > > It seems like the problem of rabbitmq, but it's running well without any > explicit error shown. Once I tried to restart the rabbitmq then instance > created successfully. Before this case happens I tried to set a 2nd external > network on Network node but failed so I rolled it back(changed configs back > and restart related services). Also checked those common CLI they're all good > like nova image-list, nova service-list,neutron agent-list ,keystone > service-list. > > Openstack-Juno, with 1 controller, 1 network, 3 compute, 1 block and 1 > object, servers have much more resources than needed. > > > controller# grep MessagingTimeout /var/log/nova/nova-api.log > ...... > 2015-11-24 15:58:00.404 18197 TRACE nova.api.openstack MessagingTimeout: > Timed out waiting for a reply to message ID 039995083875445bacdd3bd8667cda51 > 2015-11-24 15:59:02.618 18204 TRACE nova.api.openstack MessagingTimeout: > Timed out waiting for a reply to message ID 58c3ea4f3c2d43699632c1999df79e91 > 2015-11-24 18:58:28.030 18201 TRACE nova.api.openstack MessagingTimeout: > Timed out waiting for a reply to message ID 6b43bf60e2cd494fab116515c5997ab5 > > controller# grep ERROR /var/log/nova/nova-api.log > ...... > 2015-11-24 18:57:27.003 18201 ERROR oslo.messaging._drivers.impl_rabbit > [req-2b810cc0-d809-4f2c-8a01-d1f7eca62b48 ] Failed to consume message from > queue: [Errno 104] Connection reset by peer > 2015-11-24 18:58:28.030 18201 ERROR nova.api.openstack > [req-2b810cc0-d809-4f2c-8a01-d1f7eca62b48 None] Caught error: Timed out > waiting for a reply to message ID 6b43bf60e2cd494fab116515c5997ab5 > 2015-11-24 19:07:26.591 18200 ERROR oslo.messaging._drivers.impl_rabbit > [req-f532b095-5f37-432d-8cdc-06db8498b3c1 ] Failed to publish message to > topic 'conductor': [Errno 104] Connection reset by peer > 2015-11-24 19:22:36.310 18183 ERROR oslo.messaging._drivers.impl_rabbit > [req-24c4ce2a-6516-401e-a6f8-8a89758cc6fa ] Failed to publish message to > topic 'conductor': [Errno 104] Connection reset by peer > > # tail -n 80 /var/log/nova/nova-conductor.log > ...... > 2015-11-24 19:22:37.356 17770 TRACE oslo.messaging._drivers.impl_rabbit > File "/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 342, in send > 2015-11-24 19:22:37.356 17770 TRACE oslo.messaging._drivers.impl_rabbit > total_sent += fd.send(data[total_sent:], flags) > 2015-11-24 19:22:37.356 17770 TRACE oslo.messaging._drivers.impl_rabbit > error: [Errno 104] Connection reset by peer > 2015-11-24 19:22:37.356 17770 TRACE oslo.messaging._drivers.impl_rabbit > 2015-11-24 19:22:37.357 17770 INFO oslo.messaging._drivers.impl_rabbit > [req-24c4ce2a-6516-401e-a6f8-8a89758cc6fa ] Delaying reconnect for 1.0 > seconds... > 2015-11-24 19:22:38.358 17770 INFO oslo.messaging._drivers.impl_rabbit > [req-24c4ce2a-6516-401e-a6f8-8a89758cc6fa ] Connecting to AMQP server on > controller:5672 > 2015-11-24 19:22:38.379 17770 INFO oslo.messaging._drivers.impl_rabbit > [req-24c4ce2a-6516-401e-a6f8-8a89758cc6fa ] Connected to AMQP server on > controller:5672 > 2015-11-24 19:23:38.385 17770 ERROR nova.scheduler.driver > [req-24c4ce2a-6516-401e-a6f8-8a89758cc6fa None] Exception during > scheduler.run_instance > 2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver Traceback (most > recent call last): > 2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver File > "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 616, in > build_instances > 2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver request_spec, > filter_properties) > 2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver File > "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line > 49, in select_destinations > 2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver context, > request_spec, filter_properties) > 2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver File > "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line > 35, in __run_method > 2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver return > getattr(self.instance, __name)(*args, **kwargs) > 2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver File > "/usr/lib/python2.7/site-packages/nova/scheduler/client/query.py", line 34, > in select_destinations > 2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver context, > request_spec, filter_properties) > 2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver File > "/usr/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 108, in > select_destinations > 2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver > request_spec=request_spec, filter_properties=filter_properties) > 2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver File > "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 152, in > call > 2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver > retry=self.retry) > 2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver File > "/usr/lib/python2.7/site-packages/oslo/messaging/transport.py", line 90, in > _send > 2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver > timeout=timeout, retry=retry) > 2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver File > "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", > line 408, in send > 2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver retry=retry) > 2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver File > "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", > line 397, in _send > 2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver result = > self._waiter.wait(msg_id, timeout) > 2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver File > "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", > line 285, in wait > 2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver reply, ending = > self._poll_connection(msg_id, timeout) > 2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver File > "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", > line 235, in _poll_connection > 2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver % msg_id) > 2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver MessagingTimeout: > Timed out waiting for a reply to message ID d015995266a24e05b21ff0cd2ff3c6bc > 2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver > 2015-11-24 19:23:38.388 17770 WARNING nova.scheduler.driver > [req-24c4ce2a-6516-401e-a6f8-8a89758cc6fa None] [instance: > 66147c0d-fa53-4e2b-8447-d4f20ac7bcbb] Setting instance to ERROR state. > > controller# systemctl status rabbitmq-server > rabbitmq-server.service - RabbitMQ broker > Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service; enabled) > Active: active (running) since Tue 2015-11-24 15:02:41 CST; 4h 43min ago > Process: 137583 ExecStop=/usr/lib/rabbitmq/bin/rabbitmqctl stop > (code=exited, status=0/SUCCESS) > Main PID: 137639 (beam.smp) > CGroup: /system.slice/rabbitmq-server.service > ├─137639 /usr/lib64/erlang/erts-5.10.4/bin/beam.smp -W w -K true > -A30 -P 1048576 -- -root /usr/lib64/erlang -progname erl -- -home > /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_s... > ├─137654 /usr/lib64/erlang/erts-5.10.4/bin/epmd -daemon > ├─137767 inet_gethost 4 > └─137768 inet_gethost 4 > > Nov 24 15:02:40 controller systemd[1]: rabbitmq-server.service: Got > notification message from PID 137701, but reception only permitted for PID > 137639 > Nov 24 15:02:40 controller systemd[1]: rabbitmq-server.service: Got > notification message from PID 137702, but reception only permitted for PID > 137639 > Nov 24 15:02:41 controller rabbitmq-server[137639]: RabbitMQ 3.3.5. Copyright > (C) 2007-2014 GoPivotal, Inc. > Nov 24 15:02:41 controller rabbitmq-server[137639]: ## ## Licensed > under the MPL. See http://www.rabbitmq.com/ <http://www.rabbitmq.com/> > Nov 24 15:02:41 controller rabbitmq-server[137639]: ## ## > Nov 24 15:02:41 controller rabbitmq-server[137639]: ########## Logs: > /var/log/rabbitmq/[email protected] > <mailto:/var/log/rabbitmq/[email protected]> > Nov 24 15:02:41 controller rabbitmq-server[137639]: ###### ## > /var/log/rabbitmq/[email protected] > <mailto:/var/log/rabbitmq/[email protected]> > Nov 24 15:02:41 controller rabbitmq-server[137639]: ########## > Nov 24 15:02:41 controller rabbitmq-server[137639]: Starting broker... > completed with 6 plugins. > Nov 24 15:02:41 controller systemd[1]: Started RabbitMQ broker.
_______________________________________________ OpenStack-operators mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
