All, I am trying to add Neutron LBaaSv2 to a working OpenStack Liberty deployment. I am running into an issue where the lbaas agent does not appear in the output of neutron agent-list. However, the lbaas extension appears in the output of neutron ext-list. After investigating further, the lbaas-agent sends a message on the queue and times out waiting for a reply:
2016-06-06 21:09:15.958 22 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.32.20.52:5672 2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager [-] Unable to retrieve ready devices 2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager Traceback (most recent call last): 2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutro n_lbaas/services/loadbalancer/agent/agent_manager.py", line 152, in sync_state 2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager ready_instances = set(self.plugin_rpc.get_r eady_devices()) 2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutro n_lbaas/services/loadbalancer/agent/agent_api.py", line 36, in get_ready_devices 2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager return cctxt.call(self.context, 'get_ready_ devices', host=self.host) 2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo_m essaging/rpc/client.py", line 158, in call 2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager retry=self.retry) 2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo_m essaging/transport.py", line 90, in _send 2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager timeout=timeout, retry=retry) 2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo_m essaging/_drivers/amqpdriver.py", line 431, in send 2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager retry=retry) 2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo_m essaging/_drivers/amqpdriver.py", line 420, in _send 2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager result = self._waiter.wait(msg_id, timeout) 2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo_m essaging/_drivers/amqpdriver.py", line 318, in wait 2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager message = self.waiters.get(msg_id, timeout= timeout) 2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo_m essaging/_drivers/amqpdriver.py", line 223, in get 2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager 'to message ID %s' % msg_id) 2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager MessagingTimeout: Timed out waiting for a reply to message ID eae3cc1bc8614aa8ae499d92ca4ec731 I verified that the lbaas queues reside within the Rabbit cluster: bash-4.2$ rabbitmqctl list_queues n-lbaas_agent 0 n-lbaas_agent.control-server-1.novalocal 0 n-lbaas_agent.control-server-2.novalocal 0 n-lbaas_agent.control-server-3.novalocal 0 n-lbaas_agent_fanout_18a3b28c969148f3a008df8f3e5f5363 0 n-lbaas_agent_fanout_a7d48e8a1b27443d82ee4944bec44cf8 0 n-lbaas_agent_fanout_b5360edb19c240e79c71d60806977f66 0 n-lbaasv2-plugin 0 n-lbaasv2-plugin.control-server-1.novalocal 0 n-lbaasv2-plugin.control-server-2.novalocal 0 n-lbaasv2-plugin.control-server-3.novalocal 0 n-lbaasv2-plugin_fanout_5cbb6dd4fafc4c4784add8a20e0a28a5 0 n-lbaasv2-plugin_fanout_756ee4e4eee547528d0f6e3dde71b150 0 n-lbaasv2-plugin_fanout_7629f7bb85ce493d83c334dfcc2cd4aa 0 notifications.info 8 And the lbaas queues are being mirrored: # rabbitmq server logs =INFO REPORT==== 6-Jun-2016::19:01:23 === Mirrored queue 'n-lbaasv2-plugin_fanout_659b460849ef43ee834ce6d88d294b46' in vhost '/': Adding mirror on node 'rabbit@mercury-control-se rver-3': <3038.25481.1> =INFO REPORT==== 6-Jun-2016::19:01:23 === Mirrored queue 'n-lbaasv2-plugin_fanout_659b460849ef43ee834ce6d88d294b46' in vhost '/': Adding mirror on node 'rabbit@mercury-control-se rver-2': <3037.25635.1> =INFO REPORT==== 6-Jun-2016::19:01:23 === Mirrored queue 'n-lbaasv2-plugin_fanout_659b460849ef43ee834ce6d88d294b46' in vhost '/': Synchronising: 0 messages to synchronise =INFO REPORT==== 6-Jun-2016::19:01:23 === Mirrored queue 'n-lbaasv2-plugin_fanout_659b460849ef43ee834ce6d88d294b46' in vhost '/': Synchronising: all slaves already synced =INFO REPORT==== 6-Jun-2016::19:01:23 === Mirrored queue 'n-lbaasv2-plugin_fanout_659b460849ef43ee834ce6d88d294b46' in vhost '/': Synchronising: 0 messages to synchronise =INFO REPORT==== 6-Jun-2016::19:01:23 === Mirrored queue 'n-lbaasv2-plugin_fanout_659b460849ef43ee834ce6d88d294b46' in vhost '/': Synchronising: all slaves already synced ... =INFO REPORT==== 6-Jun-2016::19:01:24 === Mirrored queue 'n-lbaasv2-plugin_fanout_a6c881c1ae9c4a259b6a3f8c2f42a0b4' in vhost '/': Adding mirror on node 'rabbit@mercury-control-se rver-3': <3038.25503.1> =INFO REPORT==== 6-Jun-2016::19:01:24 === Mirrored queue 'n-lbaasv2-plugin_fanout_a6c881c1ae9c4a259b6a3f8c2f42a0b4' in vhost '/': Adding mirror on node 'rabbit@mercury-control-se rver-2': <3037.25657.1> =INFO REPORT==== 6-Jun-2016::19:01:24 === Mirrored queue 'n-lbaasv2-plugin_fanout_a6c881c1ae9c4a259b6a3f8c2f42a0b4' in vhost '/': Synchronising: 0 messages to synchronise =INFO REPORT==== 6-Jun-2016::19:01:24 === Mirrored queue 'n-lbaasv2-plugin_fanout_3d7abfb48a7a4bc8bbfb406490e9e8b6' in vhost '/': Adding mirror on node 'rabbit@mercury-control-se rver-3': <3038.25506.1> =INFO REPORT==== 6-Jun-2016::19:01:24 === Mirrored queue 'n-lbaasv2-plugin_fanout_a6c881c1ae9c4a259b6a3f8c2f42a0b4' in vhost '/': Synchronising: all slaves already synced =INFO REPORT==== 6-Jun-2016::19:01:24 === Mirrored queue 'n-lbaasv2-plugin_fanout_a6c881c1ae9c4a259b6a3f8c2f42a0b4' in vhost '/': Synchronising: 0 messages to synchronise =INFO REPORT==== 6-Jun-2016::19:01:24 === Mirrored queue 'n-lbaasv2-plugin_fanout_a6c881c1ae9c4a259b6a3f8c2f42a0b4' in vhost '/': Synchronising: all slaves already synced =INFO REPORT==== 6-Jun-2016::19:01:24 === Mirrored queue 'n-lbaasv2-plugin_fanout_3d7abfb48a7a4bc8bbfb406490e9e8b6' in vhost '/': Adding mirror on node 'rabbit@mercury-control-se rver-2': <3037.25660.1> =INFO REPORT==== 6-Jun-2016::19:01:24 === Mirrored queue 'n-lbaasv2-plugin_fanout_a6c881c1ae9c4a259b6a3f8c2f42a0b4' in vhost '/': Synchronising: 0 messages to synchronise =INFO REPORT==== 6-Jun-2016::19:01:24 === Mirrored queue 'n-lbaasv2-plugin_fanout_a6c881c1ae9c4a259b6a3f8c2f42a0b4' in vhost '/': Synchronising: all slaves already synced =INFO REPORT==== 6-Jun-2016::19:01:24 === Mirrored queue 'n-lbaasv2-plugin_fanout_3d7abfb48a7a4bc8bbfb406490e9e8b6' in vhost '/': Adding mirror on node 'rabbit@mercury-control-se rver-2': <3037.25660.1> =INFO REPORT==== 6-Jun-2016::19:01:24 === Mirrored queue 'n-lbaasv2-plugin_fanout_3d7abfb48a7a4bc8bbfb406490e9e8b6' in vhost '/': Synchronising: 0 messages to synchronise =INFO REPORT==== 6-Jun-2016::19:01:24 === Mirrored queue 'n-lbaasv2-plugin_fanout_3d7abfb48a7a4bc8bbfb406490e9e8b6' in vhost '/': Synchronising: all slaves already synced =INFO REPORT==== 6-Jun-2016::19:01:24 === Mirrored queue 'n-lbaasv2-plugin_fanout_3d7abfb48a7a4bc8bbfb406490e9e8b6' in vhost '/': Synchronising: 0 messages to synchronise =INFO REPORT==== 6-Jun-2016::19:01:24 === Mirrored queue 'n-lbaasv2-plugin_fanout_3d7abfb48a7a4bc8bbfb406490e9e8b6' in vhost '/': Synchronising: all slaves already synced All other neutron agents and openstack services are functional and communicate across the same Rabbit cluster. [1] provides additional details of the setup. It appears I may be running into bug [2]. I added myself to this bug, but it seems to have gone dormant. I would appreciate any support in resolving this issue. [1] https://gist.github.com/danehans/8691464051cb43feacac11afd4bbc447 [2] https://bugs.launchpad.net/neutron/+bug/1464241 Regards, Daneyon Hansen
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev