On Sun, May 28, 2017 at 6:48 PM Benny Zlotnik <[email protected]> wrote:
> Looking into this. Looks like all LSM tasks finished but the engine memory > lock hasn't been released yet. > Most likely the test or the sdk are not waiting correctly for the completion of the live merge. > > On Sun, May 28, 2017 at 5:17 PM, Dafna Ron <[email protected]> wrote: > >> *Hi All, * >> >> * We are seeing the same failure sporadically in three different places >> (hotunplug disk). We think this is a race condition which was introduced to >> OST with the the live storage migration test (lock on the disk is >> preventing the hotunplug). Test failed: [ 004_basic_sanity.hotunplug_disk] >> Link to suspected patches: https://gerrit.ovirt.org/#/c/77083 >> <https://gerrit.ovirt.org/#/c/77083> Link to Job: >> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6881 >> <http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6881> * >> >> >> * Link to all logs: >> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6881/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-004_basic_sanity.py/ >> <http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6881/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-004_basic_sanity.py/> >> Error snippet from the log: <error> * >> >> >> Error Message >> >> status: 409 >> reason: Conflict >> detail: Cannot hot unplug Virtual Disk. Disk vm0_disk0 is being moved or >> copied. >> >> Stacktrace >> >> Traceback (most recent call last): >> File "/usr/lib64/python2.7/unittest/case.py", line 369, in run >> testMethod() >> File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest >> self.test(*self.arg) >> File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in >> wrapped_test >> test() >> File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in >> wrapper >> return func(get_test_prefix(), *args, **kwargs) >> File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 68, in >> wrapper >> return func(prefix.virt_env.engine_vm().get_api(), *args, **kwargs) >> File >> "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/004_basic_sanity.py", >> line 498, in hotunplug_disk >> disk.deactivate() >> File >> "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line >> 31839, in deactivate >> headers={"Correlation-Id":correlation_id} >> File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", >> line 122, in request >> persistent_auth=self.__persistent_auth >> File >> "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", >> line 79, in do_request >> persistent_auth) >> File >> "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", >> line 162, in __do_request >> raise errors.RequestError(response_code, response_reason, response_body) >> RequestError: >> status: 409 >> reason: Conflict >> detail: Cannot hot unplug Virtual Disk. Disk vm0_disk0 is being moved or >> copied. >> >> >> from engine log: >> 2017-05-28 09:05:32,891-04 INFO >> [org.ovirt.engine.core.bll.storage.disk.HotUnPlugDiskFromVmCommand] (default >> task-10) [6d47ba60-4491-48ab-bb1c-fb0cdbeb76b0] Failed to Acquire Lock to >> object >> 'EngineLock:{exclusiveLocks='[9e7d8b37-2afb-4012-9bc9-8cc277ac9cdd=<DISK, >> ACTION_TYPE_FAILED_DISKS_LOCKED$diskAliases vm0_disk0>]', >> sharedLocks='[461224f5-a2af-4be2-ba1b-3563a88f398c=<VM, >> ACTION_TYPE_FAILED_VM_IS_LOCKED>]'}' >> 2017-05-28 09:05:32,891-04 WARN >> [org.ovirt.engine.core.bll.storage.disk.HotUnPlugDiskFromVmCommand] (default >> task-10) [6d47ba60-4491-48ab-bb1c-fb0cdbeb76b0] Validation of action >> 'HotUnPlugDiskFromVm' failed for user admin@internal-authz. Reasons: >> VAR__ACTION__HOT_UNPLUG,VAR__TYPE__DISK,ACTION_TYPE_FAILED_DISK_IS_BEING_MIGRATED,$DiskName >> vm0_disk0 >> 2017-05-28 09:05:32,895-04 ERROR >> [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default >> task-10) [] Operation Failed: [Cannot hot unplug Virtual Disk. Disk >> vm0_disk0 is being moved or copied.] >> 2017-05-28 09:05:33,323-04 INFO >> [org.ovirt.engine.core.bll.storage.lsm.LiveMigrateVmDisksCommand] >> (DefaultQuartzScheduler8) [99f56ceb-3c13-423f-b015-71222a8fec01] Ending >> command 'org.ovirt.engine.core.bll.storage.lsm.LiveMigrateVmDisksCommand' >> successfully. >> >> 2017-05-28 09:05:33,324-04 INFO >> [org.ovirt.engine.core.bll.storage.lsm.LiveMigrateVmDisksCommand] >> (DefaultQuartzScheduler8) [99f56ceb-3c13-423f-b015-71222a8fec01] Lock freed >> to object >> 'EngineLock:{exclusiveLocks='[9e7d8b37-2afb-4012-9bc9-8cc277ac9cdd=<DISK, >> ACTION_TYPE_FAILED_DISK_IS_BEING_MIGRATED$DiskName vm0_disk0>]', >> sharedLocks='[461224f5-a2af-4be2-ba1b-3563a88f398c=<VM, >> ACTION_TYPE_FAILED_OBJECT_LOCKED>]'}' >> >> *</error>* >> >> > _______________________________________________ > Devel mailing list > [email protected] > http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________ Devel mailing list [email protected] http://lists.ovirt.org/mailman/listinfo/devel
