[ovirt-users] Re: Delete snapshots task hung
Thank you for help Strahil, But although there where 4 images with a status 4 in the database, and did the update query on them, same bloody message, and vms wont start. Eventually, ive decided to delete the vms, and do a from scratch installation. Persistance openshift vms are still ok, so i should be able to reuse the volumes somehow. This is why sometimes a subscription is good, when a lack of knowledge on my side is happening. Production systems should not rely on upstreams unless there is a strong understanding of the product. Again, thank you so much for trying helping me out ! Cheers, Leo On Tue, Oct 15, 2019, 07:00 Leo David wrote: > Thank you Strahil, > I'll proceed with these steps and come back to you. > Cheers, > > Leo > > On Tue, Oct 15, 2019, 06:45 Strahil wrote: > >> Have you checked this thread : >> https://lists.ovirt.org/pipermail/users/2016-April/039277.html >> >> You can switch to postgre user, then 'source >> /opt/rhn/postgresql10/enable' & then 'psql engine'. >> >> As per the thread you can find illegal snapshots via '*select >> image_group_id,imagestatus from images where imagestatus =4;*' >> >> And then update them via '*update images set imagestatus =1 where >> imagestatus = 4 and ;** commit'* >> >> Best Regards, >> Strahil Nikolov >> >> On Oct 13, 2019 15:45, Leo David wrote: >> >> > >> > Hi Everyone, >> > Im still not being able to start the vms... Could anyone give me an >> advice on sorign this out ? >> > Still having th "Bad volume specification" error, although the disk is >> present on the storage. >> > This issue would force me to reinstall a 10 nodes Openshift cluster >> from scratch, which would not be so funny.. >> > Thanks, >> > >> > Leo. >> > >> > On Fri, Oct 11, 2019 at 7:12 AM Strahil wrote: >> >> >> >> >> Nah... >> >> It's done directly on the DB and I wouldn't recommend such action for >> Production Cluster. >> >> I've done it only once and it was based on some old mailing lists. >> >> >> >> Maybe someone from the dev can assist? >> >> >> >> On Oct 10, 2019 13:31, Leo David wrote: >> >> >>> >> >>> Thank you Strahil, >> >>> Could you tell me what do you mean by changing status ? Is this >> something to be done in the UI ? >> >>> >> >>> Thanks, >> >>> >> >>> Leo >> >>> >> >>> On Thu, Oct 10, 2019, 09:55 Strahil wrote: >> >> >> Maybe you can change the status of the VM in order the engine to >> know that it has to blockcommit the snapshots. >> >> Best Regards, >> Strahil Nikolov >> >> On Oct 9, 2019 09:02, Leo David wrote: >> >> > >> > Hi Everyone, >> > Please let me know if any thoughts or recommandations that could >> help me solve this issue.. >> > The real bad luck in this outage is that these 5 vms are part on an >> Openshift deployment, and now we are not able to start it up... >> > Before trying to sort this at ocp platform level by replacing the >> failed nodes with new vms, I would rather prefer to do it at the oVirt >> level and have the vms starting since the disks are still present on >> gluster. >> > Thank you so much ! >> > >> > >> > Leo >> >> > >> > >> > >> > -- >> > Best regards, Leo David >> >> ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/WYE2EO4AOCTWK4EWGMDQ7KSTF3M6JR6Q/
[ovirt-users] Re: Delete snapshots task hung
Thank you Strahil, I'll proceed with these steps and come back to you. Cheers, Leo On Tue, Oct 15, 2019, 06:45 Strahil wrote: > Have you checked this thread : > https://lists.ovirt.org/pipermail/users/2016-April/039277.html > > You can switch to postgre user, then 'source /opt/rhn/postgresql10/enable' > & then 'psql engine'. > > As per the thread you can find illegal snapshots via '*select > image_group_id,imagestatus from images where imagestatus =4;*' > > And then update them via '*update images set imagestatus =1 where > imagestatus = 4 and ;** commit'* > > Best Regards, > Strahil Nikolov > > On Oct 13, 2019 15:45, Leo David wrote: > > > > > Hi Everyone, > > Im still not being able to start the vms... Could anyone give me an > advice on sorign this out ? > > Still having th "Bad volume specification" error, although the disk is > present on the storage. > > This issue would force me to reinstall a 10 nodes Openshift cluster from > scratch, which would not be so funny.. > > Thanks, > > > > Leo. > > > > On Fri, Oct 11, 2019 at 7:12 AM Strahil wrote: > > >> > >> Nah... > >> It's done directly on the DB and I wouldn't recommend such action for > Production Cluster. > >> I've done it only once and it was based on some old mailing lists. > >> > >> Maybe someone from the dev can assist? > >> > >> On Oct 10, 2019 13:31, Leo David wrote: > > >>> > >>> Thank you Strahil, > >>> Could you tell me what do you mean by changing status ? Is this > something to be done in the UI ? > >>> > >>> Thanks, > >>> > >>> Leo > >>> > >>> On Thu, Oct 10, 2019, 09:55 Strahil wrote: > > > Maybe you can change the status of the VM in order the engine to know > that it has to blockcommit the snapshots. > > Best Regards, > Strahil Nikolov > > On Oct 9, 2019 09:02, Leo David wrote: > > > > > Hi Everyone, > > Please let me know if any thoughts or recommandations that could > help me solve this issue.. > > The real bad luck in this outage is that these 5 vms are part on an > Openshift deployment, and now we are not able to start it up... > > Before trying to sort this at ocp platform level by replacing the > failed nodes with new vms, I would rather prefer to do it at the oVirt > level and have the vms starting since the disks are still present on > gluster. > > Thank you so much ! > > > > > > Leo > > > > > > > > > -- > > Best regards, Leo David > > ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/AUXZBKKV6UNK4V5FZQFV3LRQRJIZT7EN/
[ovirt-users] Re: Delete snapshots task hung
Have you checked this thread : https://lists.ovirt.org/pipermail/users/2016-April/039277.html You can switch to postgre user, then 'source /opt/rhn/postgresql10/enable' & then 'psql engine'. As per the thread you can find illegal snapshots via 'select image_group_id,imagestatus from images where imagestatus =4;' And then update them via 'update images set imagestatus =1 where imagestatus = 4 and ; commit' Best Regards, Strahil Nikolov On Oct 13, 2019 15:45, Leo David wrote: > > Hi Everyone, > Im still not being able to start the vms... Could anyone give me an advice on > sorign this out ? > Still having th "Bad volume specification" error, although the disk is > present on the storage. > This issue would force me to reinstall a 10 nodes Openshift cluster from > scratch, which would not be so funny.. > Thanks, > > Leo. > > On Fri, Oct 11, 2019 at 7:12 AM Strahil wrote: >> >> Nah... >> It's done directly on the DB and I wouldn't recommend such action for >> Production Cluster. >> I've done it only once and it was based on some old mailing lists. >> >> Maybe someone from the dev can assist? >> >> On Oct 10, 2019 13:31, Leo David wrote: >>> >>> Thank you Strahil, >>> Could you tell me what do you mean by changing status ? Is this something >>> to be done in the UI ? >>> >>> Thanks, >>> >>> Leo >>> >>> On Thu, Oct 10, 2019, 09:55 Strahil wrote: Maybe you can change the status of the VM in order the engine to know that it has to blockcommit the snapshots. Best Regards, Strahil Nikolov On Oct 9, 2019 09:02, Leo David wrote: > > Hi Everyone, > Please let me know if any thoughts or recommandations that could help me > solve this issue.. > The real bad luck in this outage is that these 5 vms are part on an > Openshift deployment, and now we are not able to start it up... > Before trying to sort this at ocp platform level by replacing the failed > nodes with new vms, I would rather prefer to do it at the oVirt level and > have the vms starting since the disks are still present on gluster. > Thank you so much ! > > > Leo > > > > -- > Best regards, Leo David___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/VIGEIWX7VOZGLRFSWKHVSA3PPHZ3DBNT/
[ovirt-users] Re: Delete snapshots task hung
Hi Everyone, Im still not being able to start the vms... Could anyone give me an advice on sorign this out ? Still having th "Bad volume specification" error, although the disk is present on the storage. This issue would force me to reinstall a 10 nodes Openshift cluster from scratch, which would not be so funny.. Thanks, Leo. On Fri, Oct 11, 2019 at 7:12 AM Strahil wrote: > Nah... > It's done directly on the DB and I wouldn't recommend such action for > Production Cluster. > I've done it only once and it was based on some old mailing lists. > > Maybe someone from the dev can assist? > On Oct 10, 2019 13:31, Leo David wrote: > > Thank you Strahil, > Could you tell me what do you mean by changing status ? Is this something > to be done in the UI ? > > Thanks, > > Leo > > On Thu, Oct 10, 2019, 09:55 Strahil wrote: > > Maybe you can change the status of the VM in order the engine to know that > it has to blockcommit the snapshots. > > Best Regards, > Strahil Nikolov > On Oct 9, 2019 09:02, Leo David wrote: > > Hi Everyone, > Please let me know if any thoughts or recommandations that could help me > solve this issue.. > The real bad luck in this outage is that these 5 vms are part on an > Openshift deployment, and now we are not able to start it up... > Before trying to sort this at ocp platform level by replacing the failed > nodes with new vms, I would rather prefer to do it at the oVirt level and > have the vms starting since the disks are still present on gluster. > Thank you so much ! > > > Leo > > -- Best regards, Leo David ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/NFBFOA6LF3JI4CMUO66D5W2I534B5HBP/
[ovirt-users] Re: Delete snapshots task hung
Nah... It's done directly on the DB and I wouldn't recommend such action for Production Cluster. I've done it only once and it was based on some old mailing lists. Maybe someone from the dev can assist?On Oct 10, 2019 13:31, Leo David wrote: > > Thank you Strahil, > Could you tell me what do you mean by changing status ? Is this something to > be done in the UI ? > > Thanks, > > Leo > > On Thu, Oct 10, 2019, 09:55 Strahil wrote: >> >> Maybe you can change the status of the VM in order the engine to know that >> it has to blockcommit the snapshots. >> >> Best Regards, >> Strahil Nikolov >> >> On Oct 9, 2019 09:02, Leo David wrote: >>> >>> Hi Everyone, >>> Please let me know if any thoughts or recommandations that could help me >>> solve this issue.. >>> The real bad luck in this outage is that these 5 vms are part on an >>> Openshift deployment, and now we are not able to start it up... >>> Before trying to sort this at ocp platform level by replacing the failed >>> nodes with new vms, I would rather prefer to do it at the oVirt level and >>> have the vms starting since the disks are still present on gluster. >>> Thank you so much ! >>> >>> >>> Leo ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/74G24AP5CNFP4E4OYQNVIL2JAKZRRODL/
[ovirt-users] Re: Delete snapshots task hung
Thank you Strahil, Could you tell me what do you mean by changing status ? Is this something to be done in the UI ? Thanks, Leo On Thu, Oct 10, 2019, 09:55 Strahil wrote: > Maybe you can change the status of the VM in order the engine to know that > it has to blockcommit the snapshots. > > Best Regards, > Strahil Nikolov > On Oct 9, 2019 09:02, Leo David wrote: > > Hi Everyone, > Please let me know if any thoughts or recommandations that could help me > solve this issue.. > The real bad luck in this outage is that these 5 vms are part on an > Openshift deployment, and now we are not able to start it up... > Before trying to sort this at ocp platform level by replacing the failed > nodes with new vms, I would rather prefer to do it at the oVirt level and > have the vms starting since the disks are still present on gluster. > Thank you so much ! > > > Leo > > ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/SCMHKZO5EEOLNTJ7RPYGZBDE3W5C6NCT/
[ovirt-users] Re: Delete snapshots task hung
Maybe you can change the status of the VM in order the engine to know that it has to blockcommit the snapshots. Best Regards, Strahil NikolovOn Oct 9, 2019 09:02, Leo David wrote: > > Hi Everyone, > Please let me know if any thoughts or recommandations that could help me > solve this issue.. > The real bad luck in this outage is that these 5 vms are part on an Openshift > deployment, and now we are not able to start it up... > Before trying to sort this at ocp platform level by replacing the failed > nodes with new vms, I would rather prefer to do it at the oVirt level and > have the vms starting since the disks are still present on gluster. > Thank you so much ! > > > Leo ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/O3BSLRAKTPACSV3YNF5JKWDKEACZVFON/
[ovirt-users] Re: Delete snapshots task hung
Hi Everyone, Please let me know if any thoughts or recommandations that could help me solve this issue.. The real bad luck in this outage is that these 5 vms are part on an Openshift deployment, and now we are not able to start it up... Before trying to sort this at ocp platform level by replacing the failed nodes with new vms, I would rather prefer to do it at the oVirt level and have the vms starting since the disks are still present on gluster. Thank you so much ! Leo ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/MJZA5ZX4WR7QD5BFO57ATTEOQCWS3MRF/
[ovirt-users] Re: Delete snapshots task hung
Thank you Strahil,
But the vm's are not starting at all...
Error is clear: " Exit message: Bad volume specification" , but i just do
not understand how to deal with this.
Cheers,
Leo
On Tue, Oct 8, 2019 at 2:44 PM Strahil wrote:
> Try to migrate a VM from one host to another.
> I had a similar issue (1000 warnings in UI) that have stopped immediately
> I have migrated that VM.
>
> Best Regards,
> Strahil Nikolov
> On Oct 8, 2019 09:59, Leo David wrote:
>
> Hi Everyone,
> I'm waiting since 3 days for 5 x delete snapshot tasks to finish, and for
> some reason it seems to be stucked.For other vms snapshot removal took at
> most 20 mins, with havin the disks pretty much same size, and snapshots
> numbers.
> Any thoughts on how should I get this fixed ?
> Below, some lines from the engine.log, and it seems to show some
> complains regarding locks ( Failed to acquire lock and wait lock) ,
> although I am not sure if thats the root cause:
> Thank you very much !
>
> Leo
>
> 2019-10-08 09:52:48,692+03 INFO
> [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback]
> (EE-ManagedThreadFactory-engineScheduled-Thread-47)
> [73016a4a-bb2f-487f-91c5-cd027b278930] Command
> 'RemoveSnapshotSingleDiskLive' (id: '341d9c1b-2915-48d6-a8a9-9146ab19d5f8')
> waiting on child command id: '329da0fd-801b-4e0d-b7c0-fbb5c2a98bb5'
> type:'DestroyImage' to complete
> 2019-10-08 09:52:48,702+03 INFO
> [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback]
> (EE-ManagedThreadFactory-engineScheduled-Thread-47)
> [73016a4a-bb2f-487f-91c5-cd027b278930] Command
> 'RemoveSnapshotSingleDiskLive' (id: '580fa033-35fd-44f0-9979-e60e9bbf8a29')
> waiting on child command id: 'c00bdeb6-2e8b-4ef8-a3dc-1aaa088ae052'
> type:'DestroyImage' to complete
> 2019-10-08 09:52:49,713+03 INFO
> [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback]
> (EE-ManagedThreadFactory-engineScheduled-Thread-50)
> [539ba19e-0cb5-42cf-9a23-7916ee2de4a9] Command
> 'RemoveSnapshotSingleDiskLive' (id: 'de747f91-ec59-4e70-9345-77e16234bfe0')
> waiting on child command id: '10812160-cf4c-4239-bb92-1d5a847687ee'
> type:'DestroyImage' to complete
> 2019-10-08 09:52:50,725+03 INFO
> [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback]
> (EE-ManagedThreadFactory-engineScheduled-Thread-100)
> [baed2fa3-bcad-43b2-8164-480598bc72f3] Command
> 'RemoveSnapshotSingleDiskLive' (id: '4919b287-e980-4d34-a219-c08a169cd8f7')
> waiting on child command id: '5eceb6a8-f08e-42aa-8258-c907f5927e6c'
> type:'DestroyImage' to complete
> 2019-10-08 09:52:51,563+03 INFO
> [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
> (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock
> 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]',
> sharedLocks=''}'
> 2019-10-08 09:52:51,583+03 INFO
> [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
> (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock
> 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]',
> sharedLocks=''}'
> 2019-10-08 09:52:51,604+03 INFO
> [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
> (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock
> 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]',
> sharedLocks=''}'
> 2019-10-08 09:52:51,606+03 INFO
> [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
> (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock
> 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]',
> sharedLocks=''}'
> 2019-10-08 09:52:51,735+03 INFO
> [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback]
> (EE-ManagedThreadFactory-engineScheduled-Thread-94)
> [73016a4a-bb2f-487f-91c5-cd027b278930] Command 'RemoveSnapshot' (id:
> 'c9ab1344-ae27-4934-9358-d6a7b10a4f0a') waiting on child command id:
> '341d9c1b-2915-48d6-a8a9-9146ab19d5f8' type:'RemoveSnapshotSingleDiskLive'
> to complete
> 2019-10-08 09:52:52,706+03 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
> (DefaultQuartzScheduler10) [8921c9c] FINISH,
> GetGlusterLocalPhysicalVolumeListVDSCommand, return:
> [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@21830b5f,
> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@676adc3e,
> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@385a3510,
> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@af24d00,
> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@331266f2],
> log id: 1a3515fe
> 2019-10-08 09:52:52,708+03 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
> (DefaultQuartzScheduler10) [8921c9c] START,
> GetGlusterVDOVolumeListVDSCommand(HostName =
>
>
--
Best regards, Leo David
__
[ovirt-users] Re: Delete snapshots task hung
Try to migrate a VM from one host to another.
I had a similar issue (1000 warnings in UI) that have stopped immediately I
have migrated that VM.
Best Regards,
Strahil NikolovOn Oct 8, 2019 09:59, Leo David wrote:
>
> Hi Everyone,
> I'm waiting since 3 days for 5 x delete snapshot tasks to finish, and for
> some reason it seems to be stucked.For other vms snapshot removal took at
> most 20 mins, with havin the disks pretty much same size, and snapshots
> numbers.
> Any thoughts on how should I get this fixed ?
> Below, some lines from the engine.log, and it seems to show some complains
> regarding locks ( Failed to acquire lock and wait lock) , although I am not
> sure if thats the root cause:
> Thank you very much !
>
> Leo
>
> 2019-10-08 09:52:48,692+03 INFO
> [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback]
> (EE-ManagedThreadFactory-engineScheduled-Thread-47)
> [73016a4a-bb2f-487f-91c5-cd027b278930] Command 'RemoveSnapshotSingleDiskLive'
> (id: '341d9c1b-2915-48d6-a8a9-9146ab19d5f8') waiting on child command id:
> '329da0fd-801b-4e0d-b7c0-fbb5c2a98bb5' type:'DestroyImage' to complete
> 2019-10-08 09:52:48,702+03 INFO
> [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback]
> (EE-ManagedThreadFactory-engineScheduled-Thread-47)
> [73016a4a-bb2f-487f-91c5-cd027b278930] Command 'RemoveSnapshotSingleDiskLive'
> (id: '580fa033-35fd-44f0-9979-e60e9bbf8a29') waiting on child command id:
> 'c00bdeb6-2e8b-4ef8-a3dc-1aaa088ae052' type:'DestroyImage' to complete
> 2019-10-08 09:52:49,713+03 INFO
> [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback]
> (EE-ManagedThreadFactory-engineScheduled-Thread-50)
> [539ba19e-0cb5-42cf-9a23-7916ee2de4a9] Command 'RemoveSnapshotSingleDiskLive'
> (id: 'de747f91-ec59-4e70-9345-77e16234bfe0') waiting on child command id:
> '10812160-cf4c-4239-bb92-1d5a847687ee' type:'DestroyImage' to complete
> 2019-10-08 09:52:50,725+03 INFO
> [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback]
> (EE-ManagedThreadFactory-engineScheduled-Thread-100)
> [baed2fa3-bcad-43b2-8164-480598bc72f3] Command 'RemoveSnapshotSingleDiskLive'
> (id: '4919b287-e980-4d34-a219-c08a169cd8f7') waiting on child command id:
> '5eceb6a8-f08e-42aa-8258-c907f5927e6c' type:'DestroyImage' to complete
> 2019-10-08 09:52:51,563+03 INFO
> [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
> (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock
> 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]',
> sharedLocks=''}'
> 2019-10-08 09:52:51,583+03 INFO
> [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
> (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock
> 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]',
> sharedLocks=''}'
> 2019-10-08 09:52:51,604+03 INFO
> [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
> (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock
> 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]',
> sharedLocks=''}'
> 2019-10-08 09:52:51,606+03 INFO
> [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
> (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock
> 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]',
> sharedLocks=''}'
> 2019-10-08 09:52:51,735+03 INFO
> [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback]
> (EE-ManagedThreadFactory-engineScheduled-Thread-94)
> [73016a4a-bb2f-487f-91c5-cd027b278930] Command 'RemoveSnapshot' (id:
> 'c9ab1344-ae27-4934-9358-d6a7b10a4f0a') waiting on child command id:
> '341d9c1b-2915-48d6-a8a9-9146ab19d5f8' type:'RemoveSnapshotSingleDiskLive' to
> complete
> 2019-10-08 09:52:52,706+03 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
> (DefaultQuartzScheduler10) [8921c9c] FINISH,
> GetGlusterLocalPhysicalVolumeListVDSCommand, return:
> [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@21830b5f,
>
> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@676adc3e,
>
> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@385a3510,
>
> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@af24d00,
>
> org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@331266f2],
> log id: 1a3515fe
> 2019-10-08 09:52:52,708+03 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
> (DefaultQuartzScheduler10) [8921c9c] START,
> GetGlusterVDOVolumeListVDSCommand(HostName =___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Co
[ovirt-users] Re: Delete snapshots task hung
No, its 4.2.8... Thank you, Leo On Tue, Oct 8, 2019 at 1:29 PM Gianluca Cecchi wrote: > > > On Tue, Oct 8, 2019 at 12:10 PM Leo David wrote: > >> Now I'm in a worst position, after unlocking "all" entities. The tasks >> are not present anymore, the snapshots are not locked anymore, but these >> 5 vms are not able to start: >> >> >> [snip] > >> Any ideea how should I proceed to have the vms to start ? I am kind of >> stuck in this issue... >> Thank you very much in advance, >> Leo >> >> > Are you on 4.3.6? In case could it be related with the async announcement > made some hours ago? > > Gianluca > -- Best regards, Leo David ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/THDKDGDUTEBXWVTKAXFCXPZSFJAXBKTH/
[ovirt-users] Re: Delete snapshots task hung
On Tue, Oct 8, 2019 at 12:10 PM Leo David wrote: > Now I'm in a worst position, after unlocking "all" entities. The tasks > are not present anymore, the snapshots are not locked anymore, but these > 5 vms are not able to start: > > > [snip] > Any ideea how should I proceed to have the vms to start ? I am kind of > stuck in this issue... > Thank you very much in advance, > Leo > > Are you on 4.3.6? In case could it be related with the async announcement made some hours ago? Gianluca ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/6J3HJGRYJQWDKWTYXZXXOAXWXV3FJE3A/
[ovirt-users] Re: Delete snapshots task hung
Now I'm in a worst position, after unlocking "all" entities. The tasks
are not present anymore, the snapshots are not locked anymore, but these
5 vms are not able to start:
2019-10-08 12:54:25,088+03 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(ForkJoinPool-1-worker-7) [] EVENT_ID: VM_DOWN_ERROR(119), VM
openshift-04-os-infra-1 is down with error. Exit message: Bad volume
specification {'protocol': 'gluster', 'address': {'function': '0x0', 'bus':
'0x00', 'domain': '0x', 'type': 'pci', 'slot': '0x06'}, 'serial':
'eb6331b3-ec4f-4b8e-a1cd-cb763ada9f6a', 'index': 0, 'iface': 'virtio',
'apparentsize': '29933240320', 'specParams': {'pinToIoThread': '1'},
'cache': 'none', 'imageID': 'eb6331b3-ec4f-4b8e-a1cd-cb763ada9f6a',
'truesize': '30025396224', 'type': 'disk', 'domainID':
'97ced32d-bdb9-4913-a272-8a4a83ca3d1b', 'reqsize': '0', 'format': 'cow',
'poolID': 'c604b50e-2214-11e9-b449-00163e168814', 'device': 'disk', 'path':
'ssd-samsung-evo860/97ced32d-bdb9-4913-a272-8a4a83ca3d1b/images/eb6331b3-ec4f-4b8e-a1cd-cb763ada9f6a/
*f8b86437-e54c-4728-8100-ed05ef312212'*, 'propagateErrors': 'off', 'name':
'vda', 'bootOrder': '1', 'volumeID':
'f8b86437-e54c-4728-8100-ed05ef312212', 'diskType': 'network', 'alias':
'ua-eb6331b3-ec4f-4b8e-a1cd-cb763ada9f6a', 'hosts': [{'name':
'192.168.80.191', 'port': '0'}], 'discard': False}.
So i have checked the img file, it seems to be present an healthy:
# cd /rhev/data-center/mnt/glusterSD/192.168.80.191:
_ssd-samsung-evo860/97ced32d-bdb9-4913-a272-8a4a83ca3d1b/images/eb6331b3-ec4f-4b8e-a1cd-cb763ada9f6a
# qemu-img info *f8b86437-e54c-4728-8100-ed05ef312212*
image: f8b86437-e54c-4728-8100-ed05ef312212
file format: qcow2
virtual size: 50G (53687091200 bytes)
disk size: 28G
cluster_size: 65536
backing file: 4de5eae5-ab25-4a9e-a41a-2f7c2d8b272f
backing file format: qcow2
Format specific information:
compat: 1.1
lazy refcounts: false
refcount bits: 16
corrupt: false
# qemu-img check *f8b86437-e54c-4728-8100-ed05ef312212*
No errors were found on the image.
456643/819200 = 55.74% allocated, 5.36% fragmented, 0.00% compressed
clusters
Image end offset: 29933240320
# ls -l
total 62430134
-rw-rw. 1 vdsm kvm 33863434240 Oct 7 09:40
4de5eae5-ab25-4a9e-a41a-2f7c2d8b272f
-rw-rw. 1 vdsm kvm 1048576 Jun 27 13:10
4de5eae5-ab25-4a9e-a41a-2f7c2d8b272f.lease
-rw-r--r--. 1 vdsm kvm 338 Oct 6 19:01
4de5eae5-ab25-4a9e-a41a-2f7c2d8b272f.meta
-rw-rw. 1 vdsm kvm 29933240320 Oct 6 19:00
*f8b86437-e54c-4728-8100-ed05ef312212*
-rw-rw. 1 vdsm kvm 1048576 Jul 1 22:56
f8b86437-e54c-4728-8100-ed05ef312212.lease
-rw-r--r--. 1 vdsm kvm 271 Oct 6 19:01
f8b86437-e54c-4728-8100-ed05ef312212.meta
Any ideea how should I proceed to have the vms to start ? I am kind of
stuck in this issue...
Thank you very much in advance,
Leo
On Tue, Oct 8, 2019 at 9:59 AM Leo David wrote:
> Hi Everyone,
> I'm waiting since 3 days for 5 x delete snapshot tasks to finish, and for
> some reason it seems to be stucked.For other vms snapshot removal took at
> most 20 mins, with havin the disks pretty much same size, and snapshots
> numbers.
> Any thoughts on how should I get this fixed ?
> Below, some lines from the engine.log, and it seems to show some
> complains regarding locks ( Failed to acquire lock and wait lock) ,
> although I am not sure if thats the root cause:
> Thank you very much !
>
> Leo
>
> 2019-10-08 09:52:48,692+03 INFO
> [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback]
> (EE-ManagedThreadFactory-engineScheduled-Thread-47)
> [73016a4a-bb2f-487f-91c5-cd027b278930] Command
> 'RemoveSnapshotSingleDiskLive' (id: '341d9c1b-2915-48d6-a8a9-9146ab19d5f8')
> waiting on child command id: '329da0fd-801b-4e0d-b7c0-fbb5c2a98bb5'
> type:'DestroyImage' to complete
> 2019-10-08 09:52:48,702+03 INFO
> [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback]
> (EE-ManagedThreadFactory-engineScheduled-Thread-47)
> [73016a4a-bb2f-487f-91c5-cd027b278930] Command
> 'RemoveSnapshotSingleDiskLive' (id: '580fa033-35fd-44f0-9979-e60e9bbf8a29')
> waiting on child command id: 'c00bdeb6-2e8b-4ef8-a3dc-1aaa088ae052'
> type:'DestroyImage' to complete
> 2019-10-08 09:52:49,713+03 INFO
> [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback]
> (EE-ManagedThreadFactory-engineScheduled-Thread-50)
> [539ba19e-0cb5-42cf-9a23-7916ee2de4a9] Command
> 'RemoveSnapshotSingleDiskLive' (id: 'de747f91-ec59-4e70-9345-77e16234bfe0')
> waiting on child command id: '10812160-cf4c-4239-bb92-1d5a847687ee'
> type:'DestroyImage' to complete
> 2019-10-08 09:52:50,725+03 INFO
> [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback]
> (EE-ManagedThreadFactory-engineScheduled-Thread-100)
> [baed2fa3-bcad-43b2-8164-480598bc72f3] Command
> 'RemoveSnapshotSingleDiskLive' (id: '4919b287-e980-4d34-a219-c08a169cd8f7')
> waiting on child command id: '5eceb6a8-f08e-
