On Tue, Jan 6, 2015 at 5:12 PM, Sam Morrison <[email protected]> wrote:
> I’m working on upgrading our nova to Juno. > > First I upgrade the control infrastructure, conductor,api,scheduler etc. > and migrate the DB > I also set the following on the control hosts > [upgrade_levels] > compute=icehouse > > I don’t touch the nodes running nova-compute. > > Things seem to work however when I start a nova-compute I get the > following when it registers its service: > Are you restarting existing icehouse nova-compute and not juno? Can you share sanitized copies of your config file as not sure what the issue is based on the stacktrace alone. > > Line in question: > nova/virt/libvirt/driver.py > def _set_host_enabled() > service = service_obj.Service.get_by_compute_host(ctx, CONF.host) > > > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", > line 155, in call > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return > self.serializer.deserialize_entity(ctxt, result) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/nova/nova/rpc.py", line 111, in deserialize_entity > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return > self._base.deserialize_entity(context, entity) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/nova/nova/objects/base.py", line 575, in deserialize_entity > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver entity = > self._process_object(context, entity) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/nova/nova/objects/base.py", line 545, in _process_object > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver > e.kwargs['supported']) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/nova/nova/conductor/api.py", line 280, in object_backport > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return > self._manager.object_backport(context, objinst, target_version) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/nova/nova/conductor/rpcapi.py", line 435, in object_backport > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver > target_version=target_version) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", > line 155, in call > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return > self.serializer.deserialize_entity(ctxt, result) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/nova/nova/rpc.py", line 111, in deserialize_entity > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return > self._base.deserialize_entity(context, entity) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/nova/nova/objects/base.py", line 575, in deserialize_entity > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver entity = > self._process_object(context, entity) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/nova/nova/objects/base.py", line 545, in _process_object > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver > e.kwargs['supported']) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/nova/nova/conductor/api.py", line 280, in object_backport > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return > self._manager.object_backport(context, objinst, target_version) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/nova/nova/conductor/rpcapi.py", line 435, in object_backport > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver > target_version=target_version) > …. REPEATED…. > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", > line 152, in call > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver > retry=self.retry) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/transport.py", > line 90, in _send > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver > timeout=timeout, retry=retry) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", > line 436, in send > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver > retry=retry) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", > line 422, in _send > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver > retry=retry) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", > line 746, in topic_send > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver > exchange_name=exchange_name, retry=retry) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", > line 717, in publisher_send > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver > self.ensure(_error_callback, _publish, retry=retry) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", > line 617, in ensure > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver ret, > channel = autoretry_method() > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/kombu/connection.py", line > 436, in _ensured > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return > fun(*args, **kwargs) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/kombu/connection.py", line > 508, in __call__ > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return > fun(*args, channel=channels[0], **kwargs), channels[0] > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", > line 714, in _publish > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver publisher > = cls(self.conf, channel, topic=topic, **kwargs) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", > line 381, in __init__ > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver **options) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", > line 329, in __init__ > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver > self.reconnect(channel) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", > line 337, in reconnect > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver > routing_key=self.routing_key) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/kombu/messaging.py", line > 85, in __init__ > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver > self.revive(self._channel) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/kombu/messaging.py", line > 218, in revive > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver > self.declare() > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/kombu/messaging.py", line > 105, in declare > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver > self.exchange.declare() > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/kombu/entity.py", line > 166, in declare > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver > nowait=nowait, passive=passive, > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/amqp/channel.py", line > 620, in exchange_declare > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver (40, > 11), # Channel.exchange_declare_ok > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/amqp/abstract_channel.py", > line 67, in wait > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver > self.channel_id, allowed_methods) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/amqp/connection.py", line > 240, in _wait_method > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver > self.method_reader.read_method() > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/amqp/method_framing.py", > line 189, in read_method > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver raise m > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver RuntimeError: > maximum recursion depth exceeded while calling a Python object > > > In conductor I see the following (after adding a debug log message): > 2015-01-07 12:06:50.860 9781 DEBUG nova.conductor.manager > [req-2fffdbd0-fbcd-426d-adc5-9f02e1104652 None] Call to backport > Service(availability_zone=<?>,binary='nova-compute',compute_node=ComputeNode(2),created_at=2015-01-06T03:33:50Z,deleted=False,deleted_at=None,disabled=False,disabled_reason=None,host='nova-compute-i2',id=8,report_count=6911,topic='compute',updated_at=2015-01-07T01:06:10Z), > target_version=1.2 object_backport > /opt/juno/local/lib/python2.7/site-packages/nova/conductor/manager.py:439 > 2015-01-07 12:06:50.889 9781 DEBUG nova.conductor.manager > [req-2fffdbd0-fbcd-426d-adc5-9f02e1104652 None] Call to backport > Service(availability_zone=<?>,binary='nova-compute',compute_node=ComputeNode(2),created_at=2015-01-06T03:33:50Z,deleted=False,deleted_at=None,disabled=False,disabled_reason=None,host='nova-compute-i2',id=8,report_count=6911,topic='compute',updated_at=2015-01-07T01:06:10Z), > target_version=1.3 object_backport > /opt/juno/local/lib/python2.7/site-packages/nova/conductor/manager.py:439 > 2015-01-07 12:06:50.914 9781 DEBUG nova.conductor.manager > [req-2fffdbd0-fbcd-426d-adc5-9f02e1104652 None] Call to backport > Service(availability_zone=<?>,binary='nova-compute',compute_node=ComputeNode(2),created_at=2015-01-06T03:33:50Z,deleted=False,deleted_at=None,disabled=False,disabled_reason=None,host='nova-compute-i2',id=8,report_count=6911,topic='compute',updated_at=2015-01-07T01:06:10Z), > target_version=1.2 object_backport > /opt/juno/local/lib/python2.7/site-packages/nova/conductor/manager.py:439 > 2015-01-07 12:06:50.937 9781 DEBUG nova.conductor.manager > [req-2fffdbd0-fbcd-426d-adc5-9f02e1104652 None] Call to backport > Service(availability_zone=<?>,binary='nova-compute',compute_node=ComputeNode(2),created_at=2015-01-06T03:33:50Z,deleted=False,deleted_at=None,disabled=False,disabled_reason=None,host='nova-compute-i2',id=8,report_count=6911,topic='compute',updated_at=2015-01-07T01:06:10Z), > target_version=1.3 object_backport > /opt/juno/local/lib/python2.7/site-packages/nova/conductor/manager.py:439 > (repeats a bunch more times) > > What I can’t figure out is why it’s asking for a target version of 1.3, > icehouse service object is 1.2 and juno service object is 1.4 > > Anyone come across this? > We haven't seen this in any other bug reports or in our automated testing. > > Cheers, > Sam > > > > > > _______________________________________________ > Mailing list: > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > Post to : [email protected] > Unsubscribe : > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : [email protected] Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
