On 11/25/2014 8:11 AM, Sean Dague wrote:
There is currently a review stream coming into Tempest to add Cinder v2
tests in addition to the Cinder v1 tests. At the same time the currently
biggest race fail in the gate related to the projects is
http://status.openstack.org/elastic-recheck/#1373513 - which is cinder
related.

I believe these 2 facts are coupled. The number of volume tests we have
in tempest is somewhat small, and as such the likelihood of them running
simultaneously is also small. However the fact that as the # of tests
with volumes goes up we are getting more of these race fails typically
means that what's actually happening is 2 vol ops that aren't safe to
run at the same time, are.

This remains critical - https://bugs.launchpad.net/cinder/+bug/1373513 -
with no assignee.

So we really needs dedicated diving on this (last bug update with any
code was a month ago), otherwise we need to stop adding these tests to
Tempest, and honestly start skipping the volume tests if we can't have a
repeatable success.

        -Sean


I just put up an e-r query for a newly opened bug https://bugs.launchpad.net/cinder/+bug/1396186 this morning, it looks similar to bug 1373513 but without the blocked task error in syslog.

There is a three minute gap between when the volume is being deleted in c-vol logs and when we see the volume uuid logged again, at which point tempest has already timed out waiting for the delete to complete.

We should at least get some patches to add diagnostic logging in these delete flows (or periodic tasks that use the same locks/low-level i/o bound commands?) to try and pinpoint these failures.

I think I'm going to propose a skip patch for test_volume_boot_pattern since that just seems to be a never ending cause of pain until these root issues get fixed.

--

Thanks,

Matt Riedemann


_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Reply via email to