Hi David, on 2013-08-12 14:59, David Caro wrote: > On Mon 12 Aug 2013 04:13:26 AM CEST, Zhou Zheng Sheng wrote: >> Hi David, >> >> on 2013-08-09 21:45, David Caro Estevez wrote: >>> Sometimes we get this error when running the vdsm tests: >>> >>> ====================================================================== >>> ERROR: testLoopMount (mountTests.MountTests) >>> ---------------------------------------------------------------------- >>> Traceback (most recent call last): >>> File "/ephemeral0/vdsm_unit_tests_gerrit_el/tests/mountTests.py", line >>> 69, in testLoopMount >>> m.mount(mntOpts="loop") >>> File "/ephemeral0/vdsm_unit_tests_gerrit_el/vdsm/storage/mount.py", line >>> 222, in mount >>> return self._runcmd(cmd, timeout) >>> File "/ephemeral0/vdsm_unit_tests_gerrit_el/vdsm/storage/mount.py", line >>> 238, in _runcmd >>> raise MountError(rc, ";".join((out, err))) >>> MountError: (2, ';mount: could not find any free loop device\n') >>> -------------------- >> begin captured logging << -------------------- >>> Storage.Misc.excCmd: DEBUG: '/sbin/mkfs.ext2 -F /tmp/tmpq95svr' (cwd None) >>> Storage.Misc.excCmd: DEBUG: SUCCESS: <err> = 'mke2fs 1.41.12 >>> (17-May-2010)\n'; <rc> = 0 >>> Storage.Misc.excCmd: DEBUG: '/usr/bin/sudo -n /bin/mount -o loop >>> /tmp/tmpq95svr /tmp/tmpcS29EU' (cwd None) >>> >>> The problem is that it seems that the loop devices that are not being >>> released (maybe when the test fails?) and the system runs out of devices >>> eventually. >>> Can you take a look to see where the cleanup fails and fix it? >>> >> >> I think this may be related to a known bug [1] of gvfs. >> >> To workaround the bug, >> 1. Reboot Linux >> 2. See if "ps aux | grep gvfs" produces any results, then kill all gvfs >> related processes. >> 3. "cd tests", and run the tests "./run_tests_local.sh mkimageTests.py >> mountTests.py" for 10 times, and "losetup -a" should give empty result >> after each run. When gvfs is running, "losetup -a" would show 2 new loop >> devices are occupied after each run. > > gvfs is not installed on the machines, so I doubt this is the same > error :( > I suggest to add some cleanup code at the end of the tests to free all > the used loop devices (if you free only the ones you use, it will not > fail when running parallel tests). > > Adding more loop devices or cleaning them all manually is only a > temporary solution. > > Thanks!
From what I can there is 3 places would consume a loop device. In tests/parted_utils_tests.py it calls losetup directly and tears down the loop device at the end of each test. In tests/mountTests.py and tests/mkimageTests.py, it consumes loop device via loop mount, and umount the device so the loop device is freed automatically. All three places free loop device at the end of the test, unless there is exception or test fails. In this case we can catch that exception and free the device. Another reason could be other process occupying the mount point or loop device, in this case trying to free it again will fail and can not solve this problem. So I think we should investigate why it is not released and locate the bug. Could you lsof the mount point and the device to see if any process is occupying them? And could you try to run parted_utils_tests.py, mountTests.py and mkimageTests.py separately and see "losetup -a"'s report, to see which of the test module consumes but not free the device? > > -- > David Caro > > Red Hat Czech s.r.o. > Continuous Integration Engineer - EMEA ENG Virtualization R&D > > Tel.: +420 532 294 605 > Email: dc...@redhat.com > Web: www.cz.redhat.com > Red Hat Czech s.r.o., Purkyňova 99/71, 612 45, Brno, Czech Republic > RHT Global #: 82-62605 > -- Thanks and best regards! Zhou Zheng Sheng / 周征晟 E-mail: zhshz...@linux.vnet.ibm.com Telephone: 86-10-82454397 _______________________________________________ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel