Hi all, Adopted laserjetyang's suggestion, disabled DHCP agent HA (because this feature is not stable in current grizzly release and will be completed in h-3 release, we use single dhcp mode instead) and set agent_down_time to 15 secs, "no active DHCP agent" issue disappeared. But we still meet the some VM of one request cannot get IP issue. I gathered the logs and configs, need somebody help.
Configs files: https://gist.github.com/yaoyaminaco/6119120 Error logs: 1) quantum/dhcp-agent.log 2013-07-30 10:36:14 ERROR [quantum.agent.dhcp_agent] Failed reporting state! Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/quantum/agent/dhcp_agent.py", line 702, in _report_state self.agent_state) File "/usr/lib/python2.7/dist-packages/quantum/agent/rpc.py", line 66, in report_state topic=self.topic) File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/proxy.py", line 80, in call return rpc.call(context, self._get_topic(topic), msg, timeout) File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/__init__.py", line 140, in call return _get_impl().call(CONF, context, topic, msg, timeout) File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/impl_kombu.py", line 798, in call rpc_amqp.get_connection_pool(conf, Connection)) File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/amqp.py", line 613, in call rv = list(rv) File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/amqp.py", line 562, in __iter__ raise result RemoteError: Remote error: AMQPConnectionException (503, u"COMMAND_INVALID - second 'channel.open' seen", (20, 10), 'Channel.open') 2) quantum/server.log 2013-07-30 10:35:58 ERROR [quantum.openstack.common.rpc.amqp] Exception during message handling Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/amqp.py", line 436, in _process_data ctxt.reply(rval, None, connection_pool=self.connection_pool) File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/amqp.py", line 272, in reply reply, failure, ending, log_failure) File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/amqp.py", line 250, in msg_reply conn.direct_send(msg_id, rpc_common.serialize_msg(msg)) File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/amqp.py", line 152, in __exit__ self._done() File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/amqp.py", line 141, in _done self.connection.reset() File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/impl_kombu.py", line 595, in reset self.channel.close() File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 201, in close super(Channel, self).close() File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py", line 194, in close (20, 41), # Channel.close_ok File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 97, in wait return self.dispatch_method(method_sig, args, content) File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 115, in dispatch_method return amqp_method(self, args) File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py", line 273, in _close (class_id, method_id)) AMQPChannelException: (404, u"NOT_FOUND - no exchange '1bc587b773be4df8b0d956e4f02cf006' in vhost '/'", (60, 40), 'Channel.basic_publish') 2013-07-30 10:35:59 ERROR [quantum.openstack.common.rpc.common] Returning exception (404, u"NOT_FOUND - no exchange '1bc587b773be4df8b0d956e4f02cf006' in vhost '/'", (60, 40), 'Channel.basic_publish') to caller ... 2013-07-30 10:36:06 ERROR [quantum.openstack.common.rpc.amqp] Exception during message handling Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/amqp.py", line 436, in _process_data ctxt.reply(rval, None, connection_pool=self.connection_pool) File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/amqp.py", line 272, in reply reply, failure, ending, log_failure) File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/amqp.py", line 250, in msg_reply conn.direct_send(msg_id, rpc_common.serialize_msg(msg)) File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/amqp.py", line 152, in __exit__ self._done() File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/amqp.py", line 141, in _done self.connection.reset() File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/impl_kombu.py", line 596, in reset self.channel = self.connection.channel() File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 159, in channel chan = self.transport.create_channel(self.connection) File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 235, in create_channel return connection.channel() File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 144, in channel return Channel(self, channel_id) File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 183, in __init__ super(Channel, self).__init__(*args, **kwargs) File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py", line 82, in __init__ self._x_open() File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py", line 471, in _x_open (20, 11), # Channel.open_ok File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 95, in wait self.channel_id, allowed_methods) File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/connection.py", line 231, in _wait_method self.wait() File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 97, in wait return self.dispatch_method(method_sig, args, content) File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 115, in dispatch_method return amqp_method(self, args) File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/connection.py", line 380, in _close raise AMQPConnectionException(reply_code, reply_text, (class_id, method_id)) AMQPConnectionException: (503, u"COMMAND_INVALID - second 'channel.open' seen", (20, 10), 'Channel.open') 2013-07-26 20:59:43 WARNING [quantum.scheduler.dhcp_agent_scheduler] No enabled DHCP agent on host SHNetworkNode Best wishes, Yale Yuan From: Yuan, Ye A Sent: Thursday, July 25, 2013 7:58 PM To: [email protected]<mailto:[email protected]> Subject: RE: VM cannot get IP in Big cluster (grizzly, quantum, multi-host, gre) Hi all, Our team has built 15+ compute nodes Openstack cluster, and with 200+ VM running on it. But we met a stranger issue and boring me several work days. Could anyone help me ? :) The phenomenon is: if we create multi VMs in one request some or all of them cannot get IP from DHCP agent. Logs in quantum-server report "no active DHCP agent" .I read official doc and enabled DHCP HA, but all DHCP agents for network are inactive while instance boot and suddenly change to active. "no active DHCP agent" still occurs. In other words, the agent heartbeat always go active and immediately go inactive. I set agent_down_time = 120 but does not work :( My quantum server log is very big and if anyone interesting this issue , I will send it to you. Openstack is Grizzly, quantum using GRE mode, multi-host is enabled. Best wishes, Yale Yuan
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : [email protected] Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
