Re: [ovirt-users] [Users] Unable to delete a snapshot
Le 13/01/2014 08:43, Sandro Bonazzola a écrit : Il 12/01/2014 10:40, Maor Lipchuk ha scritto: Adding Sandro to the thread On 01/12/2014 11:26 AM, Maor Lipchuk wrote: On 01/09/2014 12:29 PM, Nicolas Ecarnot wrote: Hi Maor, hi everyone, Le 07/01/2014 04:09, Maor Lipchuk a écrit : Looking at bugzilla, it could be related to https://bugzilla.redhat.com/1029069 (based on the exception described at https://bugzilla.redhat.com/show_bug.cgi?id=1029069#c1) In my case, there where nothing live : the VM was shut down when creating the snapshot as well as when tyring to delete it. I understand though live storage migration sub flows are, creation of live snapshot, so it could be related. The issue there was fixed after an upgrade to 3.3.1 (as Sander mentioned it before in the mailing list) Could you give it a try and check if that works for you? I'm very shy with upgrading my oVirt production framework, but I began to read some things to upgrade it. Maybe you can lead me to a precise way to upgrade vdsm? Sandro, do you aware of any documentation which can help in upgrading ovirt, specifically VDSM? No, I'm not aware of any documentation about it. However since 3.3.1 is in ovirt-stable if you just need to update vdsm on a system I think that yum --enablerepo=ovirt-stable update vdsm* on that system should be enough. Done Also it will be great if you could open a bug on that with the full VDSM, engine logs and the list of lvs. Done : https://bugzilla.redhat.com/show_bug.cgi?id=1050901 Just to close the bug : I updated to 3.4.1-1.el6, and I can not reproduce the bug anymore. -- Nicolas Ecarnot ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Unable to delete a snapshot
Adding Sandro to the thread On 01/12/2014 11:26 AM, Maor Lipchuk wrote: On 01/09/2014 12:29 PM, Nicolas Ecarnot wrote: Hi Maor, hi everyone, Le 07/01/2014 04:09, Maor Lipchuk a écrit : Looking at bugzilla, it could be related to https://bugzilla.redhat.com/1029069 (based on the exception described at https://bugzilla.redhat.com/show_bug.cgi?id=1029069#c1) In my case, there where nothing live : the VM was shut down when creating the snapshot as well as when tyring to delete it. I understand though live storage migration sub flows are, creation of live snapshot, so it could be related. The issue there was fixed after an upgrade to 3.3.1 (as Sander mentioned it before in the mailing list) Could you give it a try and check if that works for you? I'm very shy with upgrading my oVirt production framework, but I began to read some things to upgrade it. Maybe you can lead me to a precise way to upgrade vdsm? Sandro, do you aware of any documentation which can help in upgrading ovirt, specifically VDSM? Also it will be great if you could open a bug on that with the full VDSM, engine logs and the list of lvs. Done : https://bugzilla.redhat.com/show_bug.cgi?id=1050901 Thanks ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Unable to delete a snapshot
Il 12/01/2014 10:40, Maor Lipchuk ha scritto: Adding Sandro to the thread On 01/12/2014 11:26 AM, Maor Lipchuk wrote: On 01/09/2014 12:29 PM, Nicolas Ecarnot wrote: Hi Maor, hi everyone, Le 07/01/2014 04:09, Maor Lipchuk a écrit : Looking at bugzilla, it could be related to https://bugzilla.redhat.com/1029069 (based on the exception described at https://bugzilla.redhat.com/show_bug.cgi?id=1029069#c1) In my case, there where nothing live : the VM was shut down when creating the snapshot as well as when tyring to delete it. I understand though live storage migration sub flows are, creation of live snapshot, so it could be related. The issue there was fixed after an upgrade to 3.3.1 (as Sander mentioned it before in the mailing list) Could you give it a try and check if that works for you? I'm very shy with upgrading my oVirt production framework, but I began to read some things to upgrade it. Maybe you can lead me to a precise way to upgrade vdsm? Sandro, do you aware of any documentation which can help in upgrading ovirt, specifically VDSM? No, I'm not aware of any documentation about it. However since 3.3.1 is in ovirt-stable if you just need to update vdsm on a system I think that yum --enablerepo=ovirt-stable update vdsm* on that system should be enough. CCing Dan and VDSM list for confirming Also it will be great if you could open a bug on that with the full VDSM, engine logs and the list of lvs. Done : https://bugzilla.redhat.com/show_bug.cgi?id=1050901 Thanks ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Unable to delete a snapshot
Hi Maor, hi everyone, Le 07/01/2014 04:09, Maor Lipchuk a écrit : Looking at bugzilla, it could be related to https://bugzilla.redhat.com/1029069 (based on the exception described at https://bugzilla.redhat.com/show_bug.cgi?id=1029069#c1) In my case, there where nothing live : the VM was shut down when creating the snapshot as well as when tyring to delete it. The issue there was fixed after an upgrade to 3.3.1 (as Sander mentioned it before in the mailing list) Could you give it a try and check if that works for you? I'm very shy with upgrading my oVirt production framework, but I began to read some things to upgrade it. Maybe you can lead me to a precise way to upgrade vdsm? Also it will be great if you could open a bug on that with the full VDSM, engine logs and the list of lvs. Done : https://bugzilla.redhat.com/show_bug.cgi?id=1050901 -- Nicolas Ecarnot ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Unable to delete a snapshot
self.getVolumeTag(TAG_PREFIX_IMAGE) File /usr/share/vdsm/storage/blockVolume.py, line 464, in getVolumeTag return _getVolumeTag(self.sdUUID, self.volUUID, tagPrefix) File /usr/share/vdsm/storage/blockVolume.py, line 662, in _getVolumeTag tags = lvm.getLV(sdUUID, volUUID).tags File /usr/share/vdsm/storage/lvm.py, line 851, in getLV raise se.LogicalVolumeDoesNotExistError(%s/%s % (vgName, lvName)) LogicalVolumeDoesNotExistError: Logical volume does not exist: ('11a077c7-658b-49bb-8596-a785109c24c9/_remove_me_aVmPgweS_c50561d9-c3ba-4366-b2bc-49bbfaa4cd23',) On 01/06/2014 04:39 PM, Meital Bourvine wrote: I got the attachment. This is the relevant error: 6caec3bc-fc66-42be-a642-7733fc033103::ERROR::2014-01-06 10:13:21,068::task::850::TaskManager.Task::(_setError) Task=`6caec3bc-fc66-42be-a642-7733fc033103`::Unexpected error Traceback (most recent call last): File /usr/share/vdsm/storage/task.py, line 857, in _run return fn(*args, **kargs) File /usr/share/vdsm/storage/task.py, line 318, in run return self.cmd(*self.argslist, **self.argsdict) File /usr/share/vdsm/storage/securable.py, line 68, in wrapper return f(self, *args, **kwargs) File /usr/share/vdsm/storage/sp.py, line 1937, in mergeSnapshots sdUUID, vmUUID, imgUUID, ancestor, successor, postZero) File /usr/share/vdsm/storage/image.py, line 1101, in merge dstVol = vols[ancestor] KeyError: '506085b6-40e0-4176-a4df-9102857f51f2' I don't know why it happens, so you'll have to wait for someone else to answer. - Original Message - From: Nicolas Ecarnot nico...@ecarnot.net To: users users@ovirt.org Sent: Monday, January 6, 2014 4:22:57 PM Subject: Re: [Users] Unable to delete a snapshot Le 06/01/2014 12:51, Nicolas Ecarnot a écrit : Also, Please attach the whole vdsm.log, it's hard to read it this way (lines are broken) See attachment. Actually, I don't know if this mailing list allows attachments ? -- Nicolas Ecarnot ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users -- Nicolas Ecarnot ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Unable to delete a snapshot
Le 07/01/2014 09:02, Nicolas Ecarnot a écrit : http://list-archives.org/2013/10/25/users-ovirt-org/vm-snapshot-delete-failed-iscsi-domain/f/6837397684 [...] In the URL I provided above, the op is able to lvchange -aey the device. In my case, though a lvmdiskscan + a lvs is showing me the LV, there is not device in /dev/{the proper VG}/{my missing LV}. Although I couldn't see the device in /dev, activating it with lvchange -aey and providing the complete path lead to a successful activation of the LV. Then, like in the URL above, I was able to run the VM manually from the host with a simple qemu-kvm. But when trying to run it from oVirt, I still get the same errors, and the LV gets disabled. What would be the best way now? -- Nicolas Ecarnot ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Unable to delete a snapshot
Le 07/01/2014 09:57, Nicolas Ecarnot a écrit : Le 07/01/2014 09:02, Nicolas Ecarnot a écrit : http://list-archives.org/2013/10/25/users-ovirt-org/vm-snapshot-delete-failed-iscsi-domain/f/6837397684 [...] In the URL I provided above, the op is able to lvchange -aey the device. In my case, though a lvmdiskscan + a lvs is showing me the LV, there is not device in /dev/{the proper VG}/{my missing LV}. Although I couldn't see the device in /dev, activating it with lvchange -aey and providing the complete path lead to a successful activation of the LV. Then, like in the URL above, I was able to run the VM manually from the host with a simple qemu-kvm. But when trying to run it from oVirt, I still get the same errors, and the LV gets disabled. What would be the best way now? I have to add that in this precise case, recovering the VM is way more important than to recover the snapshot, that I can live without. -- Nicolas Ecarnot ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Unable to delete a snapshot
On Tue, 2014-01-07 at 10:18 +0100, Nicolas Ecarnot wrote: Le 07/01/2014 09:57, Nicolas Ecarnot a écrit : Le 07/01/2014 09:02, Nicolas Ecarnot a écrit : http://list-archives.org/2013/10/25/users-ovirt-org/vm-snapshot-delete-failed-iscsi-domain/f/6837397684 I've sent two mails altogether - here's the other one: http://lists.ovirt.org/pipermail/users/2013-November/017557.html. In December I upgraded to Ovirt 3.3.2. After the upgrade snapshots works, but I still have a few VMs with broken snapshots. I don't need these VMs, but I'm keeping them to eventually try to repair them (when someone helps and tells me how). [...] In the URL I provided above, the op is able to lvchange -aey the device. In my case, though a lvmdiskscan + a lvs is showing me the LV, there is not device in /dev/{the proper VG}/{my missing LV}. Although I couldn't see the device in /dev, activating it with lvchange -aey and providing the complete path lead to a successful activation of the LV. Then, like in the URL above, I was able to run the VM manually from the host with a simple qemu-kvm. But when trying to run it from oVirt, I still get the same errors, and the LV gets disabled. What would be the best way now? I have to add that in this precise case, recovering the VM is way more important than to recover the snapshot, that I can live without. -- Milan Dlauhy ICZ a.s. Na hřebenech II 1718/10, 147 00 Praha 4 Nusle, CZ Tel.: +420 222 275 239 +420 724 429 878 Fax: +420 222 271 112 mailto:milan.dla...@i.cz http://www.i.cz m...@jabbim.cz, ICQ#344919380 ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[Users] Unable to delete a snapshot
Hi, With our oVirt 3.3, I created a snapshot 3 weeks ago on a VM I've properly shutdown. It ran so far. Today, after having shut it down properly, I'm trying to delete the snapshot and I get an error : Failed to delete snapshot 'blahblahbla' for VM 'myVM'. The disk is thin provisionned, accessed via virtIO, nothing special. The log below comes from the manager. I hope someone could help us because this server is quite serious. Thank you. 2014-01-06 10:10:58,826 INFO [org.ovirt.engine.core.bll.RemoveSnapshotCommand] (ajp--127.0.0.1-8702-8) Lock Acquired to object EngineLock [exclu siveLocks= key: cb953dc1-c796-457a-99a1-0e54f1c0c338 value: VM , sharedLocks= ] 2014-01-06 10:10:58,837 INFO [org.ovirt.engine.core.bll.RemoveSnapshotCommand] (ajp--127.0.0.1-8702-8) Running command: RemoveSnapshotCommand internal: false. Entities affected : ID: cb953dc1-c796-457a-99a1-0e54f1c0c338 Type: VM 2014-01-06 10:10:58,840 INFO [org.ovirt.engine.core.bll.RemoveSnapshotCommand] (ajp--127.0.0.1-8702-8) Lock freed to object EngineLock [exclusiveLocks= key: cb953dc1-c796-457a-99a1-0e54f1c0c338 value: VM , sharedLocks= ] 2014-01-06 10:10:58,844 INFO [org.ovirt.engine.core.bll.RemoveSnapshotSingleDiskCommand] (ajp--127.0.0.1-8702-8) Running command: RemoveSnapshotSingleDiskCommand internal: true. Entities affected : ID: ---- Type: Storage 2014-01-06 10:10:58,848 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MergeSnapshotsVDSCommand] (ajp--127.0.0.1-8702-8) START, MergeSnapshotsVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, storageDomainId = 11a077c7-658b-49bb-8596-a785109c24c9, imageGroupId = 69220da6-eeed-4435-aad0-7aa33f3a0d21, imageId = 506085b6-40e0-4176-a4df-9102857f51f2, imageId2 = c50561d9-c3ba-4366-b2bc-49bbfaa4cd23, vmId = cb953dc1-c796-457a-99a1-0e54f1c0c338, postZero = false), log id: 22d6503b 2014-01-06 10:10:59,511 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MergeSnapshotsVDSCommand] (ajp--127.0.0.1-8702-8) FINISH, MergeSnapshotsVDSCommand, log id: 22d6503b 2014-01-06 10:10:59,518 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (ajp--127.0.0.1-8702-8) CommandAsyncTask::Adding CommandMultiAsyncTasks object for command b402868f-b7f9-4c0e-a6fd-bdc51ff49952 2014-01-06 10:10:59,519 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (ajp--127.0.0.1-8702-8) CommandMultiAsyncTasks::AttachTask: Attaching task 6caec3bc-fc66-42be-a642-7733fc033103 to command b402868f-b7f9-4c0e-a6fd-bdc51ff49952. 2014-01-06 10:10:59,525 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (ajp--127.0.0.1-8702-8) Adding task 6caec3bc-fc66-42be-a642-7733fc033103 (Parent Command RemoveSnapshot, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. 2014-01-06 10:10:59,530 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: 3b3e6fb1, Job ID: 53867ef7-d767-45d2-b446-e5d3f5584a19, Call Stack: null, Custom Event ID: -1, Message: Snapshot 'Maj 47 60 vers 5.2.3' deletion for VM 'uc-674' was initiated by necarnot. 2014-01-06 10:10:59,532 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (ajp--127.0.0.1-8702-8) BaseAsyncTask::StartPollingTask: Starting to poll task 6caec3bc-fc66-42be-a642-7733fc033103. 2014-01-06 10:11:01,811 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (DefaultQuartzScheduler_Worker-20) Polling and updating Async Tasks: 2 tasks, 1 tasks to poll now 2014-01-06 10:11:01,824 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-20) Failed in HSMGetAllTasksStatusesVDS method 2014-01-06 10:11:01,825 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-20) Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetAllTasksStatusesVDS, error = '506085b6-40e0-4176-a4df-9102857f51f2' 2014-01-06 10:11:01,826 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (DefaultQuartzScheduler_Worker-20) SPMAsyncTask::PollTask: Polling task 6caec3bc-fc66-42be-a642-7733fc033103 (Parent Command RemoveSnapshot, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'cleanSuccess'. 2014-01-06 10:11:01,829 ERROR [org.ovirt.engine.core.bll.SPMAsyncTask] (DefaultQuartzScheduler_Worker-20) BaseAsyncTask::LogEndTaskFailure: Task 6caec3bc-fc66-42be-a642-7733fc033103 (Parent Command RemoveSnapshot, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended with failure:^M -- Result: cleanSuccess^M -- Message: VDSGenericException: VDSErrorException: Failed to HSMGetAllTasksStatusesVDS, error = '506085b6-40e0-4176-a4df-9102857f51f2',^M -- Exception: VDSGenericException: VDSErrorException: Failed to HSMGetAllTasksStatusesVDS, error =
Re: [Users] Unable to delete a snapshot
Hi Nicolas, Can u please also add the VDSM log. Thanks, Maor On 01/06/2014 11:25 AM, Nicolas Ecarnot wrote: Hi, With our oVirt 3.3, I created a snapshot 3 weeks ago on a VM I've properly shutdown. It ran so far. Today, after having shut it down properly, I'm trying to delete the snapshot and I get an error : Failed to delete snapshot 'blahblahbla' for VM 'myVM'. The disk is thin provisionned, accessed via virtIO, nothing special. The log below comes from the manager. I hope someone could help us because this server is quite serious. Thank you. 2014-01-06 10:10:58,826 INFO [org.ovirt.engine.core.bll.RemoveSnapshotCommand] (ajp--127.0.0.1-8702-8) Lock Acquired to object EngineLock [exclu siveLocks= key: cb953dc1-c796-457a-99a1-0e54f1c0c338 value: VM , sharedLocks= ] 2014-01-06 10:10:58,837 INFO [org.ovirt.engine.core.bll.RemoveSnapshotCommand] (ajp--127.0.0.1-8702-8) Running command: RemoveSnapshotCommand internal: false. Entities affected : ID: cb953dc1-c796-457a-99a1-0e54f1c0c338 Type: VM 2014-01-06 10:10:58,840 INFO [org.ovirt.engine.core.bll.RemoveSnapshotCommand] (ajp--127.0.0.1-8702-8) Lock freed to object EngineLock [exclusiveLocks= key: cb953dc1-c796-457a-99a1-0e54f1c0c338 value: VM , sharedLocks= ] 2014-01-06 10:10:58,844 INFO [org.ovirt.engine.core.bll.RemoveSnapshotSingleDiskCommand] (ajp--127.0.0.1-8702-8) Running command: RemoveSnapshotSingleDiskCommand internal: true. Entities affected : ID: ---- Type: Storage 2014-01-06 10:10:58,848 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MergeSnapshotsVDSCommand] (ajp--127.0.0.1-8702-8) START, MergeSnapshotsVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, storageDomainId = 11a077c7-658b-49bb-8596-a785109c24c9, imageGroupId = 69220da6-eeed-4435-aad0-7aa33f3a0d21, imageId = 506085b6-40e0-4176-a4df-9102857f51f2, imageId2 = c50561d9-c3ba-4366-b2bc-49bbfaa4cd23, vmId = cb953dc1-c796-457a-99a1-0e54f1c0c338, postZero = false), log id: 22d6503b 2014-01-06 10:10:59,511 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MergeSnapshotsVDSCommand] (ajp--127.0.0.1-8702-8) FINISH, MergeSnapshotsVDSCommand, log id: 22d6503b 2014-01-06 10:10:59,518 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (ajp--127.0.0.1-8702-8) CommandAsyncTask::Adding CommandMultiAsyncTasks object for command b402868f-b7f9-4c0e-a6fd-bdc51ff49952 2014-01-06 10:10:59,519 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (ajp--127.0.0.1-8702-8) CommandMultiAsyncTasks::AttachTask: Attaching task 6caec3bc-fc66-42be-a642-7733fc033103 to command b402868f-b7f9-4c0e-a6fd-bdc51ff49952. 2014-01-06 10:10:59,525 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (ajp--127.0.0.1-8702-8) Adding task 6caec3bc-fc66-42be-a642-7733fc033103 (Parent Command RemoveSnapshot, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. 2014-01-06 10:10:59,530 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: 3b3e6fb1, Job ID: 53867ef7-d767-45d2-b446-e5d3f5584a19, Call Stack: null, Custom Event ID: -1, Message: Snapshot 'Maj 47 60 vers 5.2.3' deletion for VM 'uc-674' was initiated by necarnot. 2014-01-06 10:10:59,532 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (ajp--127.0.0.1-8702-8) BaseAsyncTask::StartPollingTask: Starting to poll task 6caec3bc-fc66-42be-a642-7733fc033103. 2014-01-06 10:11:01,811 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (DefaultQuartzScheduler_Worker-20) Polling and updating Async Tasks: 2 tasks, 1 tasks to poll now 2014-01-06 10:11:01,824 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-20) Failed in HSMGetAllTasksStatusesVDS method 2014-01-06 10:11:01,825 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-20) Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetAllTasksStatusesVDS, error = '506085b6-40e0-4176-a4df-9102857f51f2' 2014-01-06 10:11:01,826 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (DefaultQuartzScheduler_Worker-20) SPMAsyncTask::PollTask: Polling task 6caec3bc-fc66-42be-a642-7733fc033103 (Parent Command RemoveSnapshot, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'cleanSuccess'. 2014-01-06 10:11:01,829 ERROR [org.ovirt.engine.core.bll.SPMAsyncTask] (DefaultQuartzScheduler_Worker-20) BaseAsyncTask::LogEndTaskFailure: Task 6caec3bc-fc66-42be-a642-7733fc033103 (Parent Command RemoveSnapshot, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended with failure:^M -- Result: cleanSuccess^M -- Message: VDSGenericException: VDSErrorException: Failed to HSMGetAllTasksStatusesVDS, error =
Re: [Users] Unable to delete a snapshot
Can you please attach vdsm.log? - Original Message - From: Nicolas Ecarnot nico...@ecarnot.net To: users users@ovirt.org Sent: Monday, January 6, 2014 11:25:15 AM Subject: [Users] Unable to delete a snapshot Hi, With our oVirt 3.3, I created a snapshot 3 weeks ago on a VM I've properly shutdown. It ran so far. Today, after having shut it down properly, I'm trying to delete the snapshot and I get an error : Failed to delete snapshot 'blahblahbla' for VM 'myVM'. The disk is thin provisionned, accessed via virtIO, nothing special. The log below comes from the manager. I hope someone could help us because this server is quite serious. Thank you. 2014-01-06 10:10:58,826 INFO [org.ovirt.engine.core.bll.RemoveSnapshotCommand] (ajp--127.0.0.1-8702-8) Lock Acquired to object EngineLock [exclu siveLocks= key: cb953dc1-c796-457a-99a1-0e54f1c0c338 value: VM , sharedLocks= ] 2014-01-06 10:10:58,837 INFO [org.ovirt.engine.core.bll.RemoveSnapshotCommand] (ajp--127.0.0.1-8702-8) Running command: RemoveSnapshotCommand internal: false. Entities affected : ID: cb953dc1-c796-457a-99a1-0e54f1c0c338 Type: VM 2014-01-06 10:10:58,840 INFO [org.ovirt.engine.core.bll.RemoveSnapshotCommand] (ajp--127.0.0.1-8702-8) Lock freed to object EngineLock [exclusiveLocks= key: cb953dc1-c796-457a-99a1-0e54f1c0c338 value: VM , sharedLocks= ] 2014-01-06 10:10:58,844 INFO [org.ovirt.engine.core.bll.RemoveSnapshotSingleDiskCommand] (ajp--127.0.0.1-8702-8) Running command: RemoveSnapshotSingleDiskCommand internal: true. Entities affected : ID: ---- Type: Storage 2014-01-06 10:10:58,848 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MergeSnapshotsVDSCommand] (ajp--127.0.0.1-8702-8) START, MergeSnapshotsVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, storageDomainId = 11a077c7-658b-49bb-8596-a785109c24c9, imageGroupId = 69220da6-eeed-4435-aad0-7aa33f3a0d21, imageId = 506085b6-40e0-4176-a4df-9102857f51f2, imageId2 = c50561d9-c3ba-4366-b2bc-49bbfaa4cd23, vmId = cb953dc1-c796-457a-99a1-0e54f1c0c338, postZero = false), log id: 22d6503b 2014-01-06 10:10:59,511 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MergeSnapshotsVDSCommand] (ajp--127.0.0.1-8702-8) FINISH, MergeSnapshotsVDSCommand, log id: 22d6503b 2014-01-06 10:10:59,518 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (ajp--127.0.0.1-8702-8) CommandAsyncTask::Adding CommandMultiAsyncTasks object for command b402868f-b7f9-4c0e-a6fd-bdc51ff49952 2014-01-06 10:10:59,519 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (ajp--127.0.0.1-8702-8) CommandMultiAsyncTasks::AttachTask: Attaching task 6caec3bc-fc66-42be-a642-7733fc033103 to command b402868f-b7f9-4c0e-a6fd-bdc51ff49952. 2014-01-06 10:10:59,525 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (ajp--127.0.0.1-8702-8) Adding task 6caec3bc-fc66-42be-a642-7733fc033103 (Parent Command RemoveSnapshot, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. 2014-01-06 10:10:59,530 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: 3b3e6fb1, Job ID: 53867ef7-d767-45d2-b446-e5d3f5584a19, Call Stack: null, Custom Event ID: -1, Message: Snapshot 'Maj 47 60 vers 5.2.3' deletion for VM 'uc-674' was initiated by necarnot. 2014-01-06 10:10:59,532 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (ajp--127.0.0.1-8702-8) BaseAsyncTask::StartPollingTask: Starting to poll task 6caec3bc-fc66-42be-a642-7733fc033103. 2014-01-06 10:11:01,811 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (DefaultQuartzScheduler_Worker-20) Polling and updating Async Tasks: 2 tasks, 1 tasks to poll now 2014-01-06 10:11:01,824 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-20) Failed in HSMGetAllTasksStatusesVDS method 2014-01-06 10:11:01,825 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-20) Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetAllTasksStatusesVDS, error = '506085b6-40e0-4176-a4df-9102857f51f2' 2014-01-06 10:11:01,826 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (DefaultQuartzScheduler_Worker-20) SPMAsyncTask::PollTask: Polling task 6caec3bc-fc66-42be-a642-7733fc033103 (Parent Command RemoveSnapshot, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'cleanSuccess'. 2014-01-06 10:11:01,829 ERROR [org.ovirt.engine.core.bll.SPMAsyncTask] (DefaultQuartzScheduler_Worker-20) BaseAsyncTask::LogEndTaskFailure: Task 6caec3bc-fc66-42be-a642-7733fc033103 (Parent Command RemoveSnapshot, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended with failure:^M -- Result: cleanSuccess^M
Re: [Users] Unable to delete a snapshot
Which subversion of oVirt was this? And which version of vdsm? I ran into the same problem with 3.3.0, snapshot deletion was fixed in 3.3.1 ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Unable to delete a snapshot
Le 06/01/2014 12:51, Nicolas Ecarnot a écrit : Also, Please attach the whole vdsm.log, it's hard to read it this way (lines are broken) See attachment. Actually, I don't know if this mailing list allows attachments ? -- Nicolas Ecarnot ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Unable to delete a snapshot
I got the attachment. This is the relevant error: 6caec3bc-fc66-42be-a642-7733fc033103::ERROR::2014-01-06 10:13:21,068::task::850::TaskManager.Task::(_setError) Task=`6caec3bc-fc66-42be-a642-7733fc033103`::Unexpected error Traceback (most recent call last): File /usr/share/vdsm/storage/task.py, line 857, in _run return fn(*args, **kargs) File /usr/share/vdsm/storage/task.py, line 318, in run return self.cmd(*self.argslist, **self.argsdict) File /usr/share/vdsm/storage/securable.py, line 68, in wrapper return f(self, *args, **kwargs) File /usr/share/vdsm/storage/sp.py, line 1937, in mergeSnapshots sdUUID, vmUUID, imgUUID, ancestor, successor, postZero) File /usr/share/vdsm/storage/image.py, line 1101, in merge dstVol = vols[ancestor] KeyError: '506085b6-40e0-4176-a4df-9102857f51f2' I don't know why it happens, so you'll have to wait for someone else to answer. - Original Message - From: Nicolas Ecarnot nico...@ecarnot.net To: users users@ovirt.org Sent: Monday, January 6, 2014 4:22:57 PM Subject: Re: [Users] Unable to delete a snapshot Le 06/01/2014 12:51, Nicolas Ecarnot a écrit : Also, Please attach the whole vdsm.log, it's hard to read it this way (lines are broken) See attachment. Actually, I don't know if this mailing list allows attachments ? -- Nicolas Ecarnot ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Unable to delete a snapshot
Hi Nicolas, I think that the initial problem started at 10:06 when VDSM tried to clear records of the ancestor volume c50561d9-c3ba-4366-b2bc-49bbfaa4cd23 (see [1]) Looking at bugzilla, it could be related to https://bugzilla.redhat.com/1029069 (based on the exception described at https://bugzilla.redhat.com/show_bug.cgi?id=1029069#c1) The issue there was fixed after an upgrade to 3.3.1 (as Sander mentioned it before in the mailing list) Could you give it a try and check if that works for you? Also it will be great if you could open a bug on that with the full VDSM, engine logs and the list of lvs. Regards, Maor [1] 236b3c5a-452a-4614-801a-c30cefbce87e::ERROR::2014-01-06 10:06:14,407::task::850::TaskManager.Task::(_setError) Task=`236b3c5a-452a-4614-801a-c30cefbce87e`::Unexpected error Traceback (most recent call last): File /usr/share/vdsm/storage/task.py, line 857, in _run return fn(*args, **kargs) File /usr/share/vdsm/storage/task.py, line 318, in run return self.cmd(*self.argslist, **self.argsdict) File /usr/share/vdsm/storage/securable.py, line 68, in wrapper return f(self, *args, **kwargs) File /usr/share/vdsm/storage/sp.py, line 1937, in mergeSnapshots sdUUID, vmUUID, imgUUID, ancestor, successor, postZero) File /usr/share/vdsm/storage/image.py, line 1162, in merge srcVol.shrinkToOptimalSize() File /usr/share/vdsm/storage/blockVolume.py, line 315, in shrinkToOptimalSize volParams = self.getVolumeParams() File /usr/share/vdsm/storage/volume.py, line 1008, in getVolumeParams volParams['imgUUID'] = self.getImage() File /usr/share/vdsm/storage/blockVolume.py, line 494, in getImage return self.getVolumeTag(TAG_PREFIX_IMAGE) File /usr/share/vdsm/storage/blockVolume.py, line 464, in getVolumeTag return _getVolumeTag(self.sdUUID, self.volUUID, tagPrefix) File /usr/share/vdsm/storage/blockVolume.py, line 662, in _getVolumeTag tags = lvm.getLV(sdUUID, volUUID).tags File /usr/share/vdsm/storage/lvm.py, line 851, in getLV raise se.LogicalVolumeDoesNotExistError(%s/%s % (vgName, lvName)) LogicalVolumeDoesNotExistError: Logical volume does not exist: ('11a077c7-658b-49bb-8596-a785109c24c9/_remove_me_aVmPgweS_c50561d9-c3ba-4366-b2bc-49bbfaa4cd23',) On 01/06/2014 04:39 PM, Meital Bourvine wrote: I got the attachment. This is the relevant error: 6caec3bc-fc66-42be-a642-7733fc033103::ERROR::2014-01-06 10:13:21,068::task::850::TaskManager.Task::(_setError) Task=`6caec3bc-fc66-42be-a642-7733fc033103`::Unexpected error Traceback (most recent call last): File /usr/share/vdsm/storage/task.py, line 857, in _run return fn(*args, **kargs) File /usr/share/vdsm/storage/task.py, line 318, in run return self.cmd(*self.argslist, **self.argsdict) File /usr/share/vdsm/storage/securable.py, line 68, in wrapper return f(self, *args, **kwargs) File /usr/share/vdsm/storage/sp.py, line 1937, in mergeSnapshots sdUUID, vmUUID, imgUUID, ancestor, successor, postZero) File /usr/share/vdsm/storage/image.py, line 1101, in merge dstVol = vols[ancestor] KeyError: '506085b6-40e0-4176-a4df-9102857f51f2' I don't know why it happens, so you'll have to wait for someone else to answer. - Original Message - From: Nicolas Ecarnot nico...@ecarnot.net To: users users@ovirt.org Sent: Monday, January 6, 2014 4:22:57 PM Subject: Re: [Users] Unable to delete a snapshot Le 06/01/2014 12:51, Nicolas Ecarnot a écrit : Also, Please attach the whole vdsm.log, it's hard to read it this way (lines are broken) See attachment. Actually, I don't know if this mailing list allows attachments ? -- Nicolas Ecarnot ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users