Hi Sam, I never did get the tests working right, but your updates to them look good. The functional piece of the patch is exactly the same as what we’re doing, though. We’ve had it running in our lab environments for a couple weeks, and I’m beginning the production upgrades today. Everything else nova-wise has been working well for us.
Thanks! Mike On 6/1/16, 9:50 PM, "Sam Morrison" <[email protected]> wrote: >Hey Mike, > >I have updated our patch for this >https://github.com/NeCTAR-RC/nova/commit/debd88652bd9f32f91ab68dd1a3979a7cdc9ec80 > >Is that what you’re doing too? > >Cheers, >Sam > > >> On 20 May 2016, at 9:26 AM, Sam Morrison <[email protected]> wrote: >> >> Ha, yeah have come across this too in our tempest runs. Haven’t had a chance >> to look into it yet. But I think you are on the right track. Just need to >> modify the tests. >> >> Cheers, >> Sam >> >> >> >>> On 20 May 2016, at 1:25 AM, Mike Dorman <[email protected]> wrote: >>> >>> We are struggling a bit with the "Send up AZ and instance_name in system >>> metadata” patch. It works fine under normal circumstances, but if there is >>> a scheduler error (No valid host was found), then there is a failure when >>> transitioning the instance to an error state, and the instance gets stuck >>> in scheduling forever: >>> >>> 2016-05-19 08:15:07.677 24589 WARNING nova.scheduler.utils >>> [req-b9e5b8b4-10bd-44d5-98d0-d89db70698bc 02b775qlx5jA >>> f48e57277a7a484290ba9afdc49a21a9 - - -] [instance: >>> 71009301-7377-49f7-9f9e-f4e718bd5c83] Setting instance to ERROR state. >>> 2016-05-19 08:15:07.679 24589 DEBUG nova.objects.instance >>> [req-b9e5b8b4-10bd-44d5-98d0-d89db70698bc 02b775qlx5jA >>> f48e57277a7a484290ba9afdc49a21a9 - - -] Lazy-loading `system_metadata' on >>> Instance uuid 71009301-7377-49f7-9f9e-f4e718bd5c83 obj_load_attr >>> /usr/lib/python2.7/site-packages/nova/objects/instance.py:873 >>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher >>> [req-b9e5b8b4-10bd-44d5-98d0-d89db70698bc 02b775qlx5jA >>> f48e57277a7a484290ba9afdc49a21a9 - - -] Exception during message handling: >>> Object action obj_load_attr failed because: attribute id not lazy-loadable >>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher Traceback >>> (most recent call last): >>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File >>> "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line >>> 142, in _dispatch_and_reply >>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher >>> executor_callback)) >>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File >>> "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line >>> 186, in _dispatch >>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher >>> executor_callback) >>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File >>> "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line >>> 129, in _do_dispatch >>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher >>> result = func(ctxt, **new_args) >>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File >>> "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 752, in >>> build_instances >>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher exc, >>> request_spec) >>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File >>> "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 625, in >>> _set_vm_state_and_notify >>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher ex, >>> request_spec, self.db) >>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File >>> "/usr/lib/python2.7/site-packages/nova/scheduler/utils.py", line 111, in >>> set_vm_state_and_notify >>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher >>> instance.save() >>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File >>> "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 213, >>> in wrapper >>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher >>> return fn(self, *args, **kwargs) >>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File >>> "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 615, in >>> save >>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher >>> self._ensure_cells_system_metadata() >>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File >>> "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 555, in >>> _ensure_cells_system_metadata >>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher >>> self.system_metadata[u'instance_name'] = self.name >>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File >>> "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 224, in >>> name >>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher >>> base_name = CONF.instance_name_template % self.id >>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File >>> "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 66, >>> in getter >>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher >>> self.obj_load_attr(name) >>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File >>> "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 864, in >>> obj_load_attr >>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher >>> reason='attribute %s not lazy-loadable' % attrname) >>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher >>> ObjectActionError: Object action obj_load_attr failed because: attribute id >>> not lazy-loadable >>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher >>> >>> >>> The problem is that _ensure_cells_system_metadata is referencing self.name >>> before the object is actually loaded from the database, so only uuid and I >>> think vm_state are set at that point. The name accessor method references >>> self.id, which (for some reason) is not lazy loadable. >>> >>> So I tried moving the _ensure_cells_system_metadata later in the save >>> method, after the object is loaded from the database (here: >>> https://github.com/openstack/nova/blob/stable/liberty/nova/objects/instance.py#L676 >>> ) That seems to work in practice, but it causes some of the tox tests to >>> fail: https://gist.github.com/misterdorm/cc7dfd235ebcc2a23009b9115b58e4d5 >>> >>> Anyways, I’m at a bit of a loss here and curious if anybody might have some >>> better insights. >>> >>> Thanks, >>> Mike >>> >>> >>> >>> From: Sam Morrison <[email protected]> >>> Date: Wednesday, May 4, 2016 at 6:23 PM >>> To: Mike Dorman <[email protected]> >>> Cc: OpenStack Operators <[email protected]> >>> Subject: Re: [Openstack-operators] Nova cells patches against Liberty >>> >>> >>> Hi Mike, >>> >>> I’ve also been working on these and have some updated patches at: >>> >>> https://github.com/NeCTAR-RC/nova/commits/stable/liberty-cellsv1 >>> >>> There are a couple of patches that you have in your tree that need updating >>> for Liberty. Mainly around supporting the v2.1 API and more things moved to >>> objects. I have also written some tests for some more of them too. I >>> haven’t tested all of these functionally >>> yet but they pass all tox tests. >>> >>> Cheers, >>> Sam >>> >>> >>> >>> >>> On 5 May 2016, at 4:19 AM, Mike Dorman <[email protected]> wrote: >>> >>> I went ahead and pulled out the Nova cells patches we’re running against >>> Liberty so that others can use them if so desired. >>> >>> https://github.com/godaddy/openstack-nova-patches >>> >>> Usual disclaimers apply here, your mileage may vary, these may not work as >>> expected in your environment, etc. We have tested these at a basic level >>> (unit tests), but are not running these for Liberty in real production yet. >>> >>> Mike >>> >>> >>> >>> >>> >>> >>> _______________________________________________ >>> OpenStack-operators mailing list >>> [email protected] >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators >>> >>> >>> >>> >>> >>> >> > _______________________________________________ OpenStack-operators mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
