Will release that when I get in to work this morning,

Needed to sleep ;)

-Josh

Kerr, Andrew wrote:
The problem is in the version of taskflow that is downloaded from pypi
by devstack. You will need to wait until a new version >0.10.0 is
available [1]

[1] https://pypi.python.org/pypi/taskflow/

Andrew Kerr
OpenStack QA
Cloud Solutions Group
NetApp

From: Bharat Kumar <bharat.kobag...@redhat.com
<mailto:bharat.kobag...@redhat.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev@lists.openstack.org
<mailto:openstack-dev@lists.openstack.org>>
Date: Friday, May 8, 2015 at 7:37 AM
To: "openstack-dev@lists.openstack.org
<mailto:openstack-dev@lists.openstack.org>"
<openstack-dev@lists.openstack.org
<mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [cinder] Taskflow 0.10.0 incompatible with
NetApp NFS drivers

GlusterFS CI job is still failing with the same issue.

I gave couple of "recheck"s on [1], after
https://review.openstack.org/#/c/181288/ patch got merged.

But still GlusterFS CI job is failing with below error [2]:
ObjectDereferencedError: Can't emit change event for attribute
'Volume.provider_location' - parent object of type <Volume> has been
garbage collected.

Also I found the same behaviour with NetApp CI also.


[1] https://review.openstack.org/#/c/165424/
[2]
http://logs.openstack.org/24/165424/6/check/check-tempest-dsvm-full-glusterfs-nv/f386477/logs/screen-c-vol.txt.gz


On 05/08/2015 10:21 AM, Joshua Harlow wrote:
Alright, it was as I had a hunch for, a small bug found in the new
algorithm to make the storage layer
copy-original,mutate-copy,save-copy,update-original (vs
update-original,save-original) more reliable.

https://bugs.launchpad.net/taskflow/+bug/1452978 opened and a one line
fix made @ https://review.openstack.org/#/c/181288/ to stop trying to
copy task results (which was activating logic that must of caused the
reference to drop out of existence and therefore the issue noted below).

Will get that released in 0.10.1 once it flushes through the pipeline.

Thanks alex for helping double check, if others want to check to
that'd be nice, can make sure that's the root cause (overzealous usage
of copy.copy, ha).

Overall I'd still *highly* recommend that the following still happen:

>> One way to get around whatever the issue is would be to change the
>> drivers to not update the object directly as it is not needed. But
>> this should not fail. Perhaps a more proper fix is for the volume
>> manager to not pass around sqlalchemy objects.

But that can be a later tweak that cinder does; using any taskflow
engine that isn't the greenthreaded/threaded/serial engine will
require results to be serializable, and therefore copyable, so that
those results can go across IPC or MQ/other boundaries. Sqlalchemy
objects won't fit either of these cases (obviously).

-Josh

Joshua Harlow wrote:
Are we sure this is taskflow? I'm wondering since those errors are more
from task code (which is in cinder) and the following seems to be a
general garbage collection issue (not connected to taskflow?):

'Exception during message handling: Can't emit change event for
attribute 'Volume.provider_location' - parent object of type <Volume>
has been garbage collected.'''

Or:

'''2015-05-07 22:42:51.142 17040 TRACE oslo_messaging.rpc.dispatcher
ObjectDereferencedError: Can't emit change event for attribute
'Volume.provider_location' - parent object of type <Volume> has been
garbage collected.'''

Alex Meade wrote:
So it seems that this will break a number of drivers, I see that
glusterfs does the same thing.

On Thu, May 7, 2015 at 10:29 PM, Alex Meade <mr.alex.me...@gmail.com
<mailto:mr.alex.me...@gmail.com>> wrote:

It appears that the release of taskflow 0.10.0 exposed an issue in
the NetApp NFS drivers. Something changed that caused the sqlalchemy
Volume object to be garbage collected even though it is passed into
create_volume()

An example error can be found in the c-vol logs here:

http://dcf901611175aa43f968-c54047c910227e27e1d6f03bb1796fd7.r95.cf5.rackcdn.com/57/181157/1/check/cinder-cDOT-NFS/0473c54/


One way to get around whatever the issue is would be to change the
drivers to not update the object directly as it is not needed. But
this should not fail. Perhaps a more proper fix is for the volume
manager to not pass around sqlalchemy objects.

+1


Something changed in taskflow, however, and we should just
understand if that has other impact.

I'd like to understand that also: the only one commit that touched this
stuff is https://github.com/openstack/taskflow/commit/227cf52 (which
basically ensured that a storage object copy is modified, then saved,
then the local object is updated vs updating the local object, and then
saving, which has problems/inconsistencies if the save fails).


-Alex


__________________________________________________________________________


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__________________________________________________________________________

OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__________________________________________________________________________

OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Warm Regards,
Bharat Kumar Kobagana
Software Engineer
OpenStack Storage – RedHat India
Mobile -+91 9949278005

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Reply via email to