Hi David,

on 2013-08-12 14:59, David Caro wrote:
> On Mon 12 Aug 2013 04:13:26 AM CEST, Zhou Zheng Sheng wrote:
>> Hi David,
>> on 2013-08-09 21:45, David Caro Estevez wrote:
>>> Sometimes we get this error when running the vdsm tests:
>>> ======================================================================
>>> ERROR: testLoopMount (mountTests.MountTests)
>>> ----------------------------------------------------------------------
>>> Traceback (most recent call last):
>>>   File "/ephemeral0/vdsm_unit_tests_gerrit_el/tests/mountTests.py", line 
>>> 69, in testLoopMount
>>>     m.mount(mntOpts="loop")
>>>   File "/ephemeral0/vdsm_unit_tests_gerrit_el/vdsm/storage/mount.py", line 
>>> 222, in mount
>>>     return self._runcmd(cmd, timeout)
>>>   File "/ephemeral0/vdsm_unit_tests_gerrit_el/vdsm/storage/mount.py", line 
>>> 238, in _runcmd
>>>     raise MountError(rc, ";".join((out, err)))
>>> MountError: (2, ';mount: could not find any free loop device\n')
>>> -------------------- >> begin captured logging << --------------------
>>> Storage.Misc.excCmd: DEBUG: '/sbin/mkfs.ext2 -F /tmp/tmpq95svr' (cwd None)
>>> Storage.Misc.excCmd: DEBUG: SUCCESS: <err> = 'mke2fs 1.41.12 
>>> (17-May-2010)\n'; <rc> = 0
>>> Storage.Misc.excCmd: DEBUG: '/usr/bin/sudo -n /bin/mount -o loop 
>>> /tmp/tmpq95svr /tmp/tmpcS29EU' (cwd None)
>>> The problem is that it seems that the loop devices that are not being 
>>> released (maybe when the test fails?) and the system runs out of devices 
>>> eventually.
>>> Can you take a look to see where the cleanup fails and fix it?
>> I think this may be related to a known bug [1] of gvfs.
>> To workaround the bug,
>> 1. Reboot Linux
>> 2. See if "ps aux | grep gvfs" produces any results, then kill all gvfs
>> related processes.
>> 3. "cd tests", and run the tests "./run_tests_local.sh mkimageTests.py
>> mountTests.py" for 10 times, and "losetup -a" should give empty result
>> after each run. When gvfs is running, "losetup -a" would show 2 new loop
>> devices are occupied after each run.
> gvfs is not installed on the machines, so I doubt this is the same 
> error :(
> I suggest to add some cleanup code at the end of the tests to free all 
> the used loop devices (if you free only the ones you use,  it will not 
> fail when running parallel tests).
> Adding more loop devices or cleaning them all manually is only a 
> temporary solution.
> Thanks!

From what I can there is 3 places would consume a loop device. In
tests/parted_utils_tests.py it calls losetup directly and tears down the
loop device at the end of each test. In tests/mountTests.py and
tests/mkimageTests.py, it consumes loop device via loop mount, and
umount the device so the loop device is freed automatically. All three
places free loop device at the end of the test, unless there is
exception or test fails. In this case we can catch that exception and
free the device.

Another reason could be other process occupying the mount point or loop
device, in this case trying to free it again will fail and can not solve
this problem. So I think we should investigate why it is not released
and locate the bug. Could you lsof the mount point and the device to see
if any process is occupying them? And could you try to run
parted_utils_tests.py, mountTests.py and mkimageTests.py separately and
see "losetup -a"'s report, to see which of the test module consumes but
not free the device?

> --
> David Caro
> Red Hat Czech s.r.o.
> Continuous Integration Engineer - EMEA ENG Virtualization R&D
> Tel.: +420 532 294 605
> Email: dc...@redhat.com
> Web: www.cz.redhat.com
> Red Hat Czech s.r.o., Purkyňova 99/71, 612 45, Brno, Czech Republic
> RHT Global #: 82-62605

Thanks and best regards!

Zhou Zheng Sheng / 周征晟
E-mail: zhshz...@linux.vnet.ibm.com
Telephone: 86-10-82454397

vdsm-devel mailing list

Reply via email to