Re: [openstack-dev] [cinder] Taskflow 0.10.0 incompatible with NetApp NFS drivers
GlusterFS CI job is still failing with the same issue. I gave couple of rechecks on [1], after https://review.openstack.org/#/c/181288/ patch got merged. But still GlusterFS CI job is failing with below error [2]: ObjectDereferencedError: Can't emit change event for attribute 'Volume.provider_location' - parent object of type Volume has been garbage collected. Also I found the same behaviour with NetApp CI also. [1] https://review.openstack.org/#/c/165424/ [2] http://logs.openstack.org/24/165424/6/check/check-tempest-dsvm-full-glusterfs-nv/f386477/logs/screen-c-vol.txt.gz On 05/08/2015 10:21 AM, Joshua Harlow wrote: Alright, it was as I had a hunch for, a small bug found in the new algorithm to make the storage layer copy-original,mutate-copy,save-copy,update-original (vs update-original,save-original) more reliable. https://bugs.launchpad.net/taskflow/+bug/1452978 opened and a one line fix made @ https://review.openstack.org/#/c/181288/ to stop trying to copy task results (which was activating logic that must of caused the reference to drop out of existence and therefore the issue noted below). Will get that released in 0.10.1 once it flushes through the pipeline. Thanks alex for helping double check, if others want to check to that'd be nice, can make sure that's the root cause (overzealous usage of copy.copy, ha). Overall I'd still *highly* recommend that the following still happen: One way to get around whatever the issue is would be to change the drivers to not update the object directly as it is not needed. But this should not fail. Perhaps a more proper fix is for the volume manager to not pass around sqlalchemy objects. But that can be a later tweak that cinder does; using any taskflow engine that isn't the greenthreaded/threaded/serial engine will require results to be serializable, and therefore copyable, so that those results can go across IPC or MQ/other boundaries. Sqlalchemy objects won't fit either of these cases (obviously). -Josh Joshua Harlow wrote: Are we sure this is taskflow? I'm wondering since those errors are more from task code (which is in cinder) and the following seems to be a general garbage collection issue (not connected to taskflow?): 'Exception during message handling: Can't emit change event for attribute 'Volume.provider_location' - parent object of type Volume has been garbage collected.''' Or: '''2015-05-07 22:42:51.142 17040 TRACE oslo_messaging.rpc.dispatcher ObjectDereferencedError: Can't emit change event for attribute 'Volume.provider_location' - parent object of type Volume has been garbage collected.''' Alex Meade wrote: So it seems that this will break a number of drivers, I see that glusterfs does the same thing. On Thu, May 7, 2015 at 10:29 PM, Alex Meade mr.alex.me...@gmail.com mailto:mr.alex.me...@gmail.com wrote: It appears that the release of taskflow 0.10.0 exposed an issue in the NetApp NFS drivers. Something changed that caused the sqlalchemy Volume object to be garbage collected even though it is passed into create_volume() An example error can be found in the c-vol logs here: http://dcf901611175aa43f968-c54047c910227e27e1d6f03bb1796fd7.r95.cf5.rackcdn.com/57/181157/1/check/cinder-cDOT-NFS/0473c54/ One way to get around whatever the issue is would be to change the drivers to not update the object directly as it is not needed. But this should not fail. Perhaps a more proper fix is for the volume manager to not pass around sqlalchemy objects. +1 Something changed in taskflow, however, and we should just understand if that has other impact. I'd like to understand that also: the only one commit that touched this stuff is https://github.com/openstack/taskflow/commit/227cf52 (which basically ensured that a storage object copy is modified, then saved, then the local object is updated vs updating the local object, and then saving, which has problems/inconsistencies if the save fails). -Alex __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Warm Regards, Bharat Kumar Kobagana Software Engineer OpenStack Storage – RedHat India Mobile - +91 9949278005
Re: [openstack-dev] [cinder] All Cinder Volume Drivers Must Have A Third Party CI by March 19, 2014
://eavesdrop.openstack.org/meetings/cinder/2014/cinder.2014-07-23-16.00.log.html#l-161 [15] - http://eavesdrop.openstack.org/meetings/cinder/2014/cinder.2014-06-18-16.02.log.html#l-255 [16] - http://eavesdrop.openstack.org/meetings/cinder/2014/cinder.2014-05-21-16.00.log.html#l-310 [17] - https://etherpad.openstack.org/p/cinder-meetup-summer-2014 [18] - http://lists.openstack.org/pipermail/openstack-dev/2014-September/045137.html [19] - http://lists.openstack.org/pipermail/openstack-dev/2014-October/047673.html [20] - http://lists.openstack.org/pipermail/openstack-dev/2014-July/039103.html [21] - http://lists.openstack.org/pipermail/openstack-dev/2014-December/051957.html [22] - http://lists.openstack.org/pipermail/openstack-dev/2014-August/043392.html [23] - http://lists.openstack.org/pipermail/openstack-dev/2014-August/042672.html [24] - http://lists.openstack.org/pipermail/openstack-dev/2014-August/041748.html [25] - http://lists.openstack.org/pipermail/openstack-dev/2014-February/026999.html [26] - http://lists.openstack.org/pipermail/openstack-dev/2014-March/028707.html [27] - http://lists.openstack.org/pipermail/openstack-dev/2014-July/039057.html [28] - http://lists.openstack.org/pipermail/openstack-dev/2014-February/027527.html [29] - http://lists.openstack.org/pipermail/openstack-dev/2014-August/041704.html [30] - https://etherpad.openstack.org/p/juno-cinder-3rd-party-cert-and-verification [31] - http://junodesignsummit.sched.org/event/56eae44976e986f39c858d784344c7d0 [32] - http://ci.openstack.org/third_party.html __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Warm Regards, Bharat Kumar Kobagana Software Engineer OpenStack Storage – RedHat India Mobile - +91 9949278005 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [tempest][glusterfs] Online Snapshot fails with GlusterFS
Hi, As part of tempest job gate-tempest-dsvm-full-glusterfs http://logs.openstack.org/11/159711/1/experimental/gate-tempest-dsvm-full-glusterfs/b2cb37e/ run [1], the test case test_snapshot_create_with_volume_in_use [2] is failing. This is because demo user is unable to create online snapshots, due to nova policy rules[3]. To avoid this issue we can modify test case, to make demo user as an admin before creating snapshot and reverting after it finishes. Another approach is to use privileged user (https://review.openstack.org/#/c/156940/) to create online snapshot. [1] http://logs.openstack.org/11/159711/1/experimental/gate-tempest-dsvm-full-glusterfs/b2cb37e/ [2] https://github.com/openstack/tempest/blob/master/tempest/api/volume/test_volumes_snapshots.py#L66 [3] https://github.com/openstack/nova/blob/master/etc/nova/policy.json#L329 -- Warm Regards, Bharat Kumar Kobagana Software Engineer OpenStack Storage – RedHat India __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [devstack] [Cinder-GlusterFS CI] centos7 gate job abrupt failures
Ran the job manually on rax VM, provided by Jeremy. (Thank you Jeremy). After running 971 test cases VM inaccessible for 569 ticks, then continues... (Look at the console.log [1]) And also have a look at dstat log. [2] The summary is: == Totals == Ran: 1125 tests in 5835. sec. - Passed: 960 - Skipped: 88 - Expected Fail: 0 - Unexpected Success: 0 - Failed: 77 Sum of execute time for each test: 13603.6755 sec. [1] https://etherpad.openstack.org/p/rax_console.txt [2] https://etherpad.openstack.org/p/rax_dstat.log On 02/24/2015 07:03 PM, Deepak Shetty wrote: FWIW, we tried to run our job in a rax provider VM (provided by ianw from his personal account) and we ran the tempest tests twice, but the OOM did not re-create. Of the 2 runs, one of the run used the same PYTHONHASHSEED as we had in one of the failed runs, still no oom. Jeremy graciously agreed to provide us 2 VMs , one each from rax and hpcloud provider to see if provider platform has anything to do with it. So we plan to run again wtih the VMs given from Jeremy , post which i will send next update here. thanx, deepak On Tue, Feb 24, 2015 at 4:50 AM, Jeremy Stanley fu...@yuggoth.org mailto:fu...@yuggoth.org wrote: Due to an image setup bug (I have a fix proposed currently), I was able to rerun this on a VM in HPCloud with 30GB memory and it completed in about an hour with a couple of tempest tests failing. Logs at: http://fungi.yuggoth.org/tmp/logs3.tar Rerunning again on another 8GB Rackspace VM with the job timeout increased to 5 hours, I was able to recreate the network connectivity issues exhibited previously. The job itself seems to have run for roughly 3 hours while failing 15 tests, and the worker was mostly unreachable for a while at the end (I don't know exactly how long) until around the time it completed. The OOM condition is present this time too according to the logs, occurring right near the end of the job. Collected logs are available at: http://fungi.yuggoth.org/tmp/logs4.tar Given the comparison between these two runs, I suspect this is either caused by memory constraints or block device I/O performance differences (or perhaps an unhappy combination of the two). Hopefully a close review of the logs will indicate which. -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Warm Regards, Bharat Kumar Kobagana Software Engineer OpenStack Storage – RedHat India Mobile - +91 9949278005 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] How to pass through devstack config
Hi, I have seen Sean Dague's patch [1], if I understood correctly, by this patch we can reduce the number of DEVSTACK_GATE variables that we need. Trying to follow this patch to configure my gate job DEVSTACK_GATE_GLUSTERFS [2]. I am not able to figure out the way to use this patch [1]. Please suggest me how to use the patch [1] to configure my gate job [2]. [1] https://review.openstack.org/#/c/145321/ [2] https://review.openstack.org/#/c/143308/7/devstack-vm-gate.sh -- Warm Regards, Bharat Kumar Kobagana Software Engineer OpenStack Storage – RedHat India __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Cinder][3rd CI] Confused about “*real* storage backend”requirement for 3rd CI.
On 01/22/2015 05:39 PM, Duncan Thomas wrote: Please take a look at https://wiki.openstack.org/wiki/Cinder/tested-3rdParty-drivers to learn how to configure devstack to use your driver rather than LVM. On 22 January 2015 at 13:28, liuxinguo liuxin...@huawei.com mailto:liuxin...@huawei.com wrote: Hi Mike, I received a email named “All Cinder Drivers Must Have a Third Party CI By March 19th 2015”and I feel confused about the “*real* storage backend”. One of the requirements is: Run Tempest [5][6] volume tests against the devstack environment that's hooked up to your *real* storage backend. ·And my confusion is: ·Every time the CI is triggered by a newly came patch, the 3rd CI will build a new devstack environment and create a default cinder.conf file whick will set the backend to “lvmdriver-1”automatically. And the tempest will run against “lvmdriver-1”. So what’s the meaning for a *real*storage backend since the cinder.conf will be set to use “lvmdriver-1”automatically for every newly came patch ? And how should I configure the cinder.conf file to run the tempest for the newly came driver patch came from different venders since different venders need different configuration for cinder.conf file and need different storage backend. I mean, does our CI should run tempest against our *real* storage backend for every newly came driver patch in cinder? Liu, Yes, by default DevStack configures cinder with LVM. But we can customize DevStack to configure cinder with our own backend (real storage backend). Below is the link to the path, enables Automatic Configuration of GlusterFS for Cinder using devstack: https://review.openstack.org/#/c/133102/ And also below it the link to Configure CEPH with Cinder using devstack: https://review.openstack.org/#/c/65113/ Above two are old way of real storage plugin implementation. Sean Dague proposed a new way of devstack plugin implementation. Have a look at below two links: https://review.openstack.org/#/c/142805/ https://review.openstack.org/#/c/142805/7/doc/source/plugins.rst Thanks and regards, Liu __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Duncan Thomas __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Warm Regards, Bharat Kumar Kobagana Software Engineer OpenStack Storage – RedHat India Mobile - +91 9949278005 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] Deploy GlusterFS server
Hi All, Regarding the patch Deploy GlusterFS Server (https://review.openstack.org/#/c/133102/). Submitted this patch long back, this patch also got Code Review +2. I think it is waiting for Workflow approval. Another task is dependent on this patch. Please review (Workflow) this patch and help me to merge this patch. -- Thanks Regards, Bharat Kumar K ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
