Hi Jay,
The tests should not be running concurrently.
We currently have 4 Natty Cloud Servers configured as unit test workers. The
machines are shared between nova/glance/keystone.
The unit test runner is pretty simple:
https://github.com/dprince/smokestack/blob/master/app/templates/unittest_runner.sh.erb
Would using a unique FAKE_FILESYSTEM_ROOTDIR for each test run or test help out
here? Perhaps prefixing each one w/ /tmp/glance-tests so that we could also add
an explicit tmp dir cleanup command in the test suite runners as well.
In the meantime I'm happy to add an explicit cleanup for /tmp/glance-tests to
make sure we have a clean slate for each test run.
Dan
-----Original Message-----
From: "Jay Pipes" <[email protected]>
Sent: Tuesday, December 13, 2011 7:14am
To: "Dan Prince" <[email protected]>
Cc: "Openstack" <[email protected]>
Subject: Re: [Openstack] unit and integration tests results for Gerrit
Thanks Dan, looks like a great start on this. Not sure what's going on
with the unit test runs in Glance, though... As an example, see:
http://smokestack.openstack.org/?go=/jobs/5600
There are dozens of errors like this:
======================================================================
ERROR: test_add_member (glance.tests.unit.test_api.TestGlanceAPI)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmp/tmp.ZaWdEoPfhP/glance_source/glance/tests/unit/test_api.py",
line 1937, in setUp
stubs.stub_out_filesystem_backend()
File "/tmp/tmp.ZaWdEoPfhP/glance_source/glance/tests/stubs.py", line
63, in stub_out_filesystem_backend
os.mkdir(FAKE_FILESYSTEM_ROOTDIR)
OSError: [Errno 17] File exists: '/tmp/glance-tests'
That FAKE_FILESYSTEM_ROOTDIR is cleaned up in the tearDown() method of
API unit tests, and these tests run perfectly fine for me locally. I
was thinking that one of the following things may be occurring:
* The unlink() of the FAKE_FILESYSTEM_ROOTDIR is not fsync'ing fast
enough, resulting in setUp()'s call to create the /tmp/glance-tests
directory is stumbling over itself
* The tests are being run in parallel somehow?
I've seen the fsync behaviour cause havoc in some of the image cache
tests before, and the solution ended up putting a small wait loop in
the test code to wait until disk buffers were flushed and a cache file
was fully removed from the filesystem. That may be happening here?
These are being run on Cloud Servers, right?
/me just trying to figure out why tests would run differently in
smokestack than Jenkins or locally...
Thanks in advance for any insight.
Cheers,
-jay
_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : [email protected]
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
