Re: [ovirt-users] Hung task finalizing live migration

[Logan Kuhn]

I had a similar situation where I was attempting and failing to delete a cinder 
disk and this fixed it. I'm using 4.0.5-5 

Regards, 
Logan 

- On Sep 10, 2016, at 9:39 AM, Gervais de Montbrun  
wrote: 

| Hi Maton,

| I have seen tasks in a weird state on my cluster also. I've had a vm get 
"stuck"
| during a migration where it says "migrating to" in the web GUI, but it has
| finished migrating hours ago... If I click "Cancel Migraton" the gui tells me
| that it is not migrating, but I can't do any action on the vm because I am 
then
| told that the vm can't be acted upon while it is migrating. I also try to kill
| the task, but there are none listed

| What has worked for me has been to put my hosted-engine in global maintenance
| mode, then ssh into the hosted engine and run the "engine-setup" command. I am
| not saying the is the best course of action, but when the engine comes back
| online the task is cleared.

| Cheers,
| Gervais

|| On Sep 10, 2016, at 11:06 AM, Maton, Brett < mat...@ltresources.co.uk > 
wrote:

|| Anyone know how to fix this broken task ?

|| It's persisted through a reboot of all hosts and the engine, something needs
|| deleting from the database to clear the task and release the locked disk

|| On 8 September 2016 at 13:25, Maton, Brett < mat...@ltresources.co.uk > 
wrote:

||| Thanks for the pointer Mikhail, however I don't get any tasks listed with 
that
||| command:

||| vdsClient -s 0 getAllTasksStatuses

||| /usr/share/vdsm/vdsClient.py:33: DeprecationWarning: vdscli uses xmlrpc. 
since
||| ovirt 3.6 xmlrpc is deprecated, please use vdsm.jsonrpcvdscli
||| from vdsm import utils, vdscli, constants

||| {'status': {'message': 'OK', 'code': 0}, 'allTasksStatus': {}}

||| On 8 September 2016 at 09:51, Краснобаев Михаил < mi...@ya.ru > wrote:

 Hi,
 There is a way to cancel a running task - look here
 http://lists.ovirt.org/pipermail/users/2014-November/028946.html
 I was able to stop snapshot deletion this way.
 Best, Mikhail.
 08.09.2016, 08:14, "Maton, Brett" < mat...@ltresources.co.uk >:

| Any suggestions ?

| THe task has been hung for 5 days now, I can't start the machine or 
destroy it.

| On 7 September 2016 at 06:49, Maton, Brett < mat...@ltresources.co.uk > 
wrote:

|| Sorry just hit reply

|| I'm seeing these errors in the logs which look related to the problem:

|| 2016-09-07 06 :46:35,123 ERROR
|| [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller]
|| (DefaultQuartzScheduler6) [19c58c0d] Failed invoking callback end method
|| 'onFailed' for command ' 07608003 -ca05-4e2e-b917-85ce525c011b' with 
exception
|| 'null', the callback is marked for end method retries
|| 2016-09-07 06 :46:45,184 ERROR [ org.ovirt.engine.core.bll.Com 
mandsFactory]
|| (DefaultQuartzScheduler7) [19c58c0d] Error in invocating CTOR of command
|| 'LiveMigrateDisk': null
|| 2016-09-07 06 :46:45,185 ERROR
|| [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller]
|| (DefaultQuartzScheduler7) [19c58c0d] Failed invoking callback end method
|| 'onFailed' for command ' 07608003 -ca05-4e2e-b917-85ce525c011b' with 
exception
|| 'null', the callback is marked for end method retries

|| On 5 September 2016 at 06:46, Nir Soffer < nsof...@redhat.com > wrote:

||| Hi Maton,

||| Please reply to the list, not to me directly.

||| Ala, can you look at this? is this a known issue?

||| Thanks,
||| Nir

||| On Mon, Sep 5, 2016 at 8:43 AM, Maton, Brett < mat...@ltresources.co.uk 
> wrote:
||| > Log files as requested

||| > https://ufile.io/4fc35 vdsm log
||| > https://ufile.io/e9836 engine 03-Sep
||| > https://ufile.io/15f37 engine 04-Sep

||| > vdsm log stops on the 01-Sep...

||| > Couple of entries from the event log:

||| > Sep 3, 2016 7:31:07 PM Snapshot 'Auto-generated for Live Storage
||| > Migration' deletion for VM 'lv01' has been completed.
||| > Sep 3, 2016 6:46:46 PM Snapshot 'Auto-generated for Live Storage
||| > Migration' deletion for VM 'lv01' was initiated by SYSTEM

||| > And the related tasks

||| > Removing Snapshot Auto-generated for Live Storage Migration of VM lv01
||| > Sep 3, 2016 6:46:44 PM N/A 29f45ca9
||| > Validating Sep 3, 2016 6:46:44 PM until Sep 3, 2016 6:46:44 PM
||| > Executing Sep 3, 2016 6:46:44 PM until Sep 3, 2016 7:31:06 PM

||| > Finalizing Sep 3, 2016 7:31:06 PM N/A



||| > On 4 September 2016 at 14:27, Nir Soffer < nsof...@redhat.com > wrote:

||| >> On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett < 
mat...@ltresources.co.uk >
||| >> wrote:

||| >>> How do I fix / kill a hung vdsm task?

||| >>> It seems to have completed the task but is stuck finalising.

||| >>> Removing Snapshot Auto-generated for Live Storage Migration
||| >>> Validating
||| >>> Executing
||| >>> (hour glass) Finalizing

||| >>> Task has been 'stuck' finalising

Re: [ovirt-users] Hung task finalizing live migration

[Gervais de Montbrun]

YAY!! Glad it worked for you.

:-)

Cheers,
Gervais



> On Sep 10, 2016, at 12:05 PM, Maton, Brett  wrote:
> 
> Way-hey! finally the task has gone and I can do 'stuff' with that VM again.
> 
> Thanks Gervais, you're a star
> 
> On 10 September 2016 at 15:40, Maton, Brett  > wrote:
> Thanks Gervais I'll give that a go
> 
> On 10 September 2016 at 15:39, Gervais de Montbrun  > wrote:
> Hi Maton,
> 
> I have seen tasks in a weird state on my cluster also. I've had a vm get 
> "stuck" during a migration where it says "migrating to" in the web GUI, but 
> it has finished migrating hours ago... If I click "Cancel Migraton" the gui 
> tells me that it is not migrating, but I can't do any action on the vm 
> because I am then told that the vm can't be acted upon while it is migrating. 
> I also try to kill the task, but there are none listed
> 
> What has worked for me has been to put my hosted-engine in global maintenance 
> mode, then ssh into the hosted engine and run the "engine-setup" command. I 
> am not saying the is the best course of action, but when the engine comes 
> back online the task is cleared.
> 
> Cheers,
> Gervais
> 
> 
> 
>> On Sep 10, 2016, at 11:06 AM, Maton, Brett > > wrote:
>> 
>> Anyone know how to fix this broken task ?
>> 
>> It's persisted through a reboot of all hosts and the engine, something needs 
>> deleting from the database to clear the task and release the locked disk
>> 
>> On 8 September 2016 at 13:25, Maton, Brett > > wrote:
>> Thanks for the pointer Mikhail, however I don't get any tasks listed with 
>> that command:
>> 
>> vdsClient -s 0 getAllTasksStatuses
>> 
>> /usr/share/vdsm/vdsClient.py:33: DeprecationWarning: vdscli uses xmlrpc. 
>> since ovirt 3.6 xmlrpc is deprecated, please use vdsm.jsonrpcvdscli
>>   from vdsm import utils, vdscli, constants
>> 
>> {'status': {'message': 'OK', 'code': 0}, 'allTasksStatus': {}}
>> 
>> 
>> On 8 September 2016 at 09:51, Краснобаев Михаил > > wrote:
>> Hi,
>>  
>> There is a way to cancel a running task  -  look here 
>> http://lists.ovirt.org/pipermail/users/2014-November/028946.html 
>> 
>> I was able to stop snapshot deletion this way.
>>  
>> Best, Mikhail.
>>  
>> 08.09.2016, 08:14, "Maton, Brett" > >:
>>> Any suggestions ?
>>> 
>>> THe task has been hung for 5 days now, I can't start the machine or destroy 
>>> it.
>>> 
>>> 
>>> On 7 September 2016 at 06:49, Maton, Brett >> > wrote:
>>> Sorry just hit reply
>>> 
>>> I'm seeing these errors in the logs which look related to the problem:
>>> 
>>> 
>>> 2016-09-07 06:46:35,123 ERROR 
>>> [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] 
>>> (DefaultQuartzScheduler6) [19c58c0d] Failed invoking callback end method 
>>> 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with 
>>> exception 'null', the callback is marked for end method retries
>>> 2016-09-07 06:46:45,184 ERROR [org.ovirt.engine.core.bll.Com 
>>> mandsFactory] 
>>> (DefaultQuartzScheduler7) [19c58c0d] Error in invocating CTOR of command 
>>> 'LiveMigrateDisk': null
>>> 2016-09-07 06:46:45,185 ERROR 
>>> [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] 
>>> (DefaultQuartzScheduler7) [19c58c0d] Failed invoking callback end method 
>>> 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with 
>>> exception 'null', the callback is marked for end method retries
>>> 
>>> On 5 September 2016 at 06:46, Nir Soffer >> > wrote:
>>> Hi Maton,
>>> 
>>> Please reply to the list, not to me directly.
>>> 
>>> Ala, can you look at this? is this a known issue?
>>> 
>>> Thanks,
>>> Nir
>>> 
>>> On Mon, Sep 5, 2016 at 8:43 AM, Maton, Brett >> > wrote:
>>> > Log files as requested
>>> >
>>> > https://ufile.io/4fc35  vdsm log
>>> > https://ufile.io/e9836  engine 03-Sep
>>> > https://ufile.io/15f37  engine 04-Sep
>>> >
>>> > vdsm log stops on the 01-Sep...
>>> >
>>> > Couple of entries from the event log:
>>> >
>>> > Sep 3, 2016 7:31:07 PMSnapshot 'Auto-generated for Live Storage
>>> > Migration' deletion for VM 'lv01' has been completed.
>>> > Sep 3, 2016 6:46:46 PMSnapshot 'Auto-generated for Live Storage
>>> > Migration' deletion for VM 'lv01' was initiated by SYSTEM
>>> >
>>> > And the related tasks
>>> >
>>> > Removing Snapshot Auto-generated for Live Storage Migration of VM lv01
>>> > Sep 3, 2016 6:46:44 PMN/A29f45ca9
>>> > ValidatingSep 3, 2016 6:46:44 PMuntilSep 3, 2016 6:46:44 PM
>>> > ExecutingSep 3, 2016 6:46:44 PMuntilSep 3, 2016 7:31:06 PM
>>> >
>>> > FinalizingSep 3, 2016 7:31:06 PMN/A
>

Re: [ovirt-users] Hung task finalizing live migration

[Maton, Brett]

Way-hey! finally the task has gone and I can do 'stuff' with that VM again.

Thanks Gervais, you're a star

On 10 September 2016 at 15:40, Maton, Brett 
wrote:

> Thanks Gervais I'll give that a go
>
> On 10 September 2016 at 15:39, Gervais de Montbrun  > wrote:
>
>> Hi Maton,
>>
>> I have seen tasks in a weird state on my cluster also. I've had a vm get
>> "stuck" during a migration where it says "migrating to" in the web GUI, but
>> it has finished migrating hours ago... If I click "Cancel Migraton" the gui
>> tells me that it is not migrating, but I can't do any action on the vm
>> because I am then told that the vm can't be acted upon while it is
>> migrating. I also try to kill the task, but there are none listed
>>
>> What has worked for me has been to put my hosted-engine in global
>> maintenance mode, then ssh into the hosted engine and run the
>> "engine-setup" command. I am not saying the is the best course of action,
>> but when the engine comes back online the task is cleared.
>>
>> Cheers,
>> Gervais
>>
>>
>>
>> On Sep 10, 2016, at 11:06 AM, Maton, Brett 
>> wrote:
>>
>> Anyone know how to fix this broken task ?
>>
>> It's persisted through a reboot of all hosts and the engine, something
>> needs deleting from the database to clear the task and release the locked
>> disk
>>
>> On 8 September 2016 at 13:25, Maton, Brett 
>> wrote:
>>
>>> Thanks for the pointer Mikhail, however I don't get any tasks listed
>>> with that command:
>>>
>>> vdsClient -s 0 getAllTasksStatuses
>>>
>>> /usr/share/vdsm/vdsClient.py:33: DeprecationWarning: vdscli uses
>>> xmlrpc. since ovirt 3.6 xmlrpc is deprecated, please use vdsm.jsonrpcvdscli
>>>   from vdsm import utils, vdscli, constants
>>>
>>> {'status': {'message': 'OK', 'code': 0}, 'allTasksStatus': {}}
>>>
>>>
>>> On 8 September 2016 at 09:51, Краснобаев Михаил  wrote:
>>>
 Hi,

 There is a way to cancel a running task  -  look here
 http://lists.ovirt.org/pipermail/users/2014-November/028946.html
 I was able to stop snapshot deletion this way.

 Best, Mikhail.

 08.09.2016, 08:14, "Maton, Brett" :

 Any suggestions ?

 THe task has been hung for 5 days now, I can't start the machine or
 destroy it.


 On 7 September 2016 at 06:49, Maton, Brett 
 wrote:

 Sorry just hit reply

 I'm seeing these errors in the logs which look related to the problem:


 2016-09-07 06:46:35,123 ERROR 
 [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller]
 (DefaultQuartzScheduler6) [19c58c0d] Failed invoking callback end method
 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with
 exception 'null', the callback is marked for end method retries
 2016-09-07 06:46:45,184 ERROR [org.ovirt.engine.core.bll.Com
 mandsFactory]
 (DefaultQuartzScheduler7) [19c58c0d] Error in invocating CTOR of command
 'LiveMigrateDisk': null
 2016-09-07 06:46:45,185 ERROR 
 [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller]
 (DefaultQuartzScheduler7) [19c58c0d] Failed invoking callback end method
 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with
 exception 'null', the callback is marked for end method retries

 On 5 September 2016 at 06:46, Nir Soffer  wrote:

 Hi Maton,

 Please reply to the list, not to me directly.

 Ala, can you look at this? is this a known issue?

 Thanks,
 Nir

 On Mon, Sep 5, 2016 at 8:43 AM, Maton, Brett 
 wrote:
 > Log files as requested
 >
 > https://ufile.io/4fc35 vdsm log
 > https://ufile.io/e9836 engine 03-Sep
 > https://ufile.io/15f37 engine 04-Sep
 >
 > vdsm log stops on the 01-Sep...
 >
 > Couple of entries from the event log:
 >
 > Sep 3, 2016 7:31:07 PMSnapshot 'Auto-generated for Live Storage
 > Migration' deletion for VM 'lv01' has been completed.
 > Sep 3, 2016 6:46:46 PMSnapshot 'Auto-generated for Live Storage
 > Migration' deletion for VM 'lv01' was initiated by SYSTEM
 >
 > And the related tasks
 >
 > Removing Snapshot Auto-generated for Live Storage Migration of VM lv01
 > Sep 3, 2016 6:46:44 PMN/A29f45ca9
 > ValidatingSep 3, 2016 6:46:44 PMuntilSep 3, 2016 6:46:44
 PM
 > ExecutingSep 3, 2016 6:46:44 PMuntilSep 3, 2016 7:31:06 PM
 >
 > FinalizingSep 3, 2016 7:31:06 PMN/A
 >
 >
 >
 > On 4 September 2016 at 14:27, Nir Soffer  wrote:
 >>
 >> On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett <
 mat...@ltresources.co.uk>
 >> wrote:
 >>>
 >>> How do I fix / kill a hung vdsm task?
 >>>
 >>> It seems to have completed the task but is stuck finalising.
 >>>
 >>> Removing Snapshot Auto-generated for Live Storage Migration
 >>> Validating
 >>> Executing
 >>> (hour glass) Fin

Re: [ovirt-users] Hung task finalizing live migration

[Maton, Brett]

Thanks Gervais I'll give that a go

On 10 September 2016 at 15:39, Gervais de Montbrun 
wrote:

> Hi Maton,
>
> I have seen tasks in a weird state on my cluster also. I've had a vm get
> "stuck" during a migration where it says "migrating to" in the web GUI, but
> it has finished migrating hours ago... If I click "Cancel Migraton" the gui
> tells me that it is not migrating, but I can't do any action on the vm
> because I am then told that the vm can't be acted upon while it is
> migrating. I also try to kill the task, but there are none listed
>
> What has worked for me has been to put my hosted-engine in global
> maintenance mode, then ssh into the hosted engine and run the
> "engine-setup" command. I am not saying the is the best course of action,
> but when the engine comes back online the task is cleared.
>
> Cheers,
> Gervais
>
>
>
> On Sep 10, 2016, at 11:06 AM, Maton, Brett 
> wrote:
>
> Anyone know how to fix this broken task ?
>
> It's persisted through a reboot of all hosts and the engine, something
> needs deleting from the database to clear the task and release the locked
> disk
>
> On 8 September 2016 at 13:25, Maton, Brett 
> wrote:
>
>> Thanks for the pointer Mikhail, however I don't get any tasks listed with
>> that command:
>>
>> vdsClient -s 0 getAllTasksStatuses
>>
>> /usr/share/vdsm/vdsClient.py:33: DeprecationWarning: vdscli uses xmlrpc.
>> since ovirt 3.6 xmlrpc is deprecated, please use vdsm.jsonrpcvdscli
>>   from vdsm import utils, vdscli, constants
>>
>> {'status': {'message': 'OK', 'code': 0}, 'allTasksStatus': {}}
>>
>>
>> On 8 September 2016 at 09:51, Краснобаев Михаил  wrote:
>>
>>> Hi,
>>>
>>> There is a way to cancel a running task  -  look here
>>> http://lists.ovirt.org/pipermail/users/2014-November/028946.html
>>> I was able to stop snapshot deletion this way.
>>>
>>> Best, Mikhail.
>>>
>>> 08.09.2016, 08:14, "Maton, Brett" :
>>>
>>> Any suggestions ?
>>>
>>> THe task has been hung for 5 days now, I can't start the machine or
>>> destroy it.
>>>
>>>
>>> On 7 September 2016 at 06:49, Maton, Brett 
>>> wrote:
>>>
>>> Sorry just hit reply
>>>
>>> I'm seeing these errors in the logs which look related to the problem:
>>>
>>>
>>> 2016-09-07 06:46:35,123 ERROR 
>>> [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller]
>>> (DefaultQuartzScheduler6) [19c58c0d] Failed invoking callback end method
>>> 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with
>>> exception 'null', the callback is marked for end method retries
>>> 2016-09-07 06:46:45,184 ERROR [org.ovirt.engine.core.bll.Com
>>> mandsFactory]
>>> (DefaultQuartzScheduler7) [19c58c0d] Error in invocating CTOR of command
>>> 'LiveMigrateDisk': null
>>> 2016-09-07 06:46:45,185 ERROR 
>>> [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller]
>>> (DefaultQuartzScheduler7) [19c58c0d] Failed invoking callback end method
>>> 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with
>>> exception 'null', the callback is marked for end method retries
>>>
>>> On 5 September 2016 at 06:46, Nir Soffer  wrote:
>>>
>>> Hi Maton,
>>>
>>> Please reply to the list, not to me directly.
>>>
>>> Ala, can you look at this? is this a known issue?
>>>
>>> Thanks,
>>> Nir
>>>
>>> On Mon, Sep 5, 2016 at 8:43 AM, Maton, Brett 
>>> wrote:
>>> > Log files as requested
>>> >
>>> > https://ufile.io/4fc35 vdsm log
>>> > https://ufile.io/e9836 engine 03-Sep
>>> > https://ufile.io/15f37 engine 04-Sep
>>> >
>>> > vdsm log stops on the 01-Sep...
>>> >
>>> > Couple of entries from the event log:
>>> >
>>> > Sep 3, 2016 7:31:07 PMSnapshot 'Auto-generated for Live Storage
>>> > Migration' deletion for VM 'lv01' has been completed.
>>> > Sep 3, 2016 6:46:46 PMSnapshot 'Auto-generated for Live Storage
>>> > Migration' deletion for VM 'lv01' was initiated by SYSTEM
>>> >
>>> > And the related tasks
>>> >
>>> > Removing Snapshot Auto-generated for Live Storage Migration of VM lv01
>>> > Sep 3, 2016 6:46:44 PMN/A29f45ca9
>>> > ValidatingSep 3, 2016 6:46:44 PMuntilSep 3, 2016 6:46:44 PM
>>> > ExecutingSep 3, 2016 6:46:44 PMuntilSep 3, 2016 7:31:06 PM
>>> >
>>> > FinalizingSep 3, 2016 7:31:06 PMN/A
>>> >
>>> >
>>> >
>>> > On 4 September 2016 at 14:27, Nir Soffer  wrote:
>>> >>
>>> >> On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett <
>>> mat...@ltresources.co.uk>
>>> >> wrote:
>>> >>>
>>> >>> How do I fix / kill a hung vdsm task?
>>> >>>
>>> >>> It seems to have completed the task but is stuck finalising.
>>> >>>
>>> >>> Removing Snapshot Auto-generated for Live Storage Migration
>>> >>> Validating
>>> >>> Executing
>>> >>> (hour glass) Finalizing
>>> >>>
>>> >>> Task has been 'stuck' finalising for over 13 hours
>>> >>
>>> >>
>>> >> Can you share engine and vdsm logs since the time the merge was
>>> started?
>>> >>
>>> >> Nir
>>> >
>>> >
>>>
>>> ,
>>>
>>> ___
>>> Users mailing list
>>> Users@ovirt.org

Re: [ovirt-users] Hung task finalizing live migration

[Gervais de Montbrun]

Hi Maton,

I have seen tasks in a weird state on my cluster also. I've had a vm get 
"stuck" during a migration where it says "migrating to" in the web GUI, but it 
has finished migrating hours ago... If I click "Cancel Migraton" the gui tells 
me that it is not migrating, but I can't do any action on the vm because I am 
then told that the vm can't be acted upon while it is migrating. I also try to 
kill the task, but there are none listed

What has worked for me has been to put my hosted-engine in global maintenance 
mode, then ssh into the hosted engine and run the "engine-setup" command. I am 
not saying the is the best course of action, but when the engine comes back 
online the task is cleared.

Cheers,
Gervais



> On Sep 10, 2016, at 11:06 AM, Maton, Brett  wrote:
> 
> Anyone know how to fix this broken task ?
> 
> It's persisted through a reboot of all hosts and the engine, something needs 
> deleting from the database to clear the task and release the locked disk
> 
> On 8 September 2016 at 13:25, Maton, Brett  > wrote:
> Thanks for the pointer Mikhail, however I don't get any tasks listed with 
> that command:
> 
> vdsClient -s 0 getAllTasksStatuses
> 
> /usr/share/vdsm/vdsClient.py:33: DeprecationWarning: vdscli uses xmlrpc. 
> since ovirt 3.6 xmlrpc is deprecated, please use vdsm.jsonrpcvdscli
>   from vdsm import utils, vdscli, constants
> 
> {'status': {'message': 'OK', 'code': 0}, 'allTasksStatus': {}}
> 
> 
> On 8 September 2016 at 09:51, Краснобаев Михаил  > wrote:
> Hi,
>  
> There is a way to cancel a running task  -  look here 
> http://lists.ovirt.org/pipermail/users/2014-November/028946.html 
> 
> I was able to stop snapshot deletion this way.
>  
> Best, Mikhail.
>  
> 08.09.2016, 08:14, "Maton, Brett"  >:
>> Any suggestions ?
>> 
>> THe task has been hung for 5 days now, I can't start the machine or destroy 
>> it.
>> 
>> 
>> On 7 September 2016 at 06:49, Maton, Brett > > wrote:
>> Sorry just hit reply
>> 
>> I'm seeing these errors in the logs which look related to the problem:
>> 
>> 
>> 2016-09-07 06:46:35,123 ERROR 
>> [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] 
>> (DefaultQuartzScheduler6) [19c58c0d] Failed invoking callback end method 
>> 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with exception 
>> 'null', the callback is marked for end method retries
>> 2016-09-07 06:46:45,184 ERROR [org.ovirt.engine.core.bll.Com 
>> mandsFactory] 
>> (DefaultQuartzScheduler7) [19c58c0d] Error in invocating CTOR of command 
>> 'LiveMigrateDisk': null
>> 2016-09-07 06:46:45,185 ERROR 
>> [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] 
>> (DefaultQuartzScheduler7) [19c58c0d] Failed invoking callback end method 
>> 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with exception 
>> 'null', the callback is marked for end method retries
>> 
>> On 5 September 2016 at 06:46, Nir Soffer > > wrote:
>> Hi Maton,
>> 
>> Please reply to the list, not to me directly.
>> 
>> Ala, can you look at this? is this a known issue?
>> 
>> Thanks,
>> Nir
>> 
>> On Mon, Sep 5, 2016 at 8:43 AM, Maton, Brett > > wrote:
>> > Log files as requested
>> >
>> > https://ufile.io/4fc35  vdsm log
>> > https://ufile.io/e9836  engine 03-Sep
>> > https://ufile.io/15f37  engine 04-Sep
>> >
>> > vdsm log stops on the 01-Sep...
>> >
>> > Couple of entries from the event log:
>> >
>> > Sep 3, 2016 7:31:07 PMSnapshot 'Auto-generated for Live Storage
>> > Migration' deletion for VM 'lv01' has been completed.
>> > Sep 3, 2016 6:46:46 PMSnapshot 'Auto-generated for Live Storage
>> > Migration' deletion for VM 'lv01' was initiated by SYSTEM
>> >
>> > And the related tasks
>> >
>> > Removing Snapshot Auto-generated for Live Storage Migration of VM lv01
>> > Sep 3, 2016 6:46:44 PMN/A29f45ca9
>> > ValidatingSep 3, 2016 6:46:44 PMuntilSep 3, 2016 6:46:44 PM
>> > ExecutingSep 3, 2016 6:46:44 PMuntilSep 3, 2016 7:31:06 PM
>> >
>> > FinalizingSep 3, 2016 7:31:06 PMN/A
>> >
>> >
>> >
>> > On 4 September 2016 at 14:27, Nir Soffer > > > wrote:
>> >>
>> >> On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett > >> >
>> >> wrote:
>> >>>
>> >>> How do I fix / kill a hung vdsm task?
>> >>>
>> >>> It seems to have completed the task but is stuck finalising.
>> >>>
>> >>> Removing Snapshot Auto-generated for Live Storage Migration
>> >>> Validating
>> >>> Executing
>> >>> (hour glass) Finalizing
>> >>>
>> >>> Task has been 'stuck' finalising for over 13 hours
>> >>
>> >>
>> >> Can you share engine and vdsm logs since the time the merge w

Re: [ovirt-users] Hung task finalizing live migration

[Maton, Brett]

Anyone know how to fix this broken task ?

It's persisted through a reboot of all hosts and the engine, something
needs deleting from the database to clear the task and release the locked
disk

On 8 September 2016 at 13:25, Maton, Brett  wrote:

> Thanks for the pointer Mikhail, however I don't get any tasks listed with
> that command:
>
> vdsClient -s 0 getAllTasksStatuses
>
> /usr/share/vdsm/vdsClient.py:33: DeprecationWarning: vdscli uses xmlrpc.
> since ovirt 3.6 xmlrpc is deprecated, please use vdsm.jsonrpcvdscli
>   from vdsm import utils, vdscli, constants
>
> {'status': {'message': 'OK', 'code': 0}, 'allTasksStatus': {}}
>
>
> On 8 September 2016 at 09:51, Краснобаев Михаил  wrote:
>
>> Hi,
>>
>> There is a way to cancel a running task  -  look here
>> http://lists.ovirt.org/pipermail/users/2014-November/028946.html
>> I was able to stop snapshot deletion this way.
>>
>> Best, Mikhail.
>>
>> 08.09.2016, 08:14, "Maton, Brett" :
>>
>> Any suggestions ?
>>
>> THe task has been hung for 5 days now, I can't start the machine or
>> destroy it.
>>
>>
>> On 7 September 2016 at 06:49, Maton, Brett 
>> wrote:
>>
>> Sorry just hit reply
>>
>> I'm seeing these errors in the logs which look related to the problem:
>>
>>
>> 2016-09-07 06:46:35,123 ERROR 
>> [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller]
>> (DefaultQuartzScheduler6) [19c58c0d] Failed invoking callback end method
>> 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with
>> exception 'null', the callback is marked for end method retries
>> 2016-09-07 06:46:45,184 ERROR [org.ovirt.engine.core.bll.CommandsFactory]
>> (DefaultQuartzScheduler7) [19c58c0d] Error in invocating CTOR of command
>> 'LiveMigrateDisk': null
>> 2016-09-07 06:46:45,185 ERROR 
>> [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller]
>> (DefaultQuartzScheduler7) [19c58c0d] Failed invoking callback end method
>> 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with
>> exception 'null', the callback is marked for end method retries
>>
>> On 5 September 2016 at 06:46, Nir Soffer  wrote:
>>
>> Hi Maton,
>>
>> Please reply to the list, not to me directly.
>>
>> Ala, can you look at this? is this a known issue?
>>
>> Thanks,
>> Nir
>>
>> On Mon, Sep 5, 2016 at 8:43 AM, Maton, Brett 
>> wrote:
>> > Log files as requested
>> >
>> > https://ufile.io/4fc35 vdsm log
>> > https://ufile.io/e9836 engine 03-Sep
>> > https://ufile.io/15f37 engine 04-Sep
>> >
>> > vdsm log stops on the 01-Sep...
>> >
>> > Couple of entries from the event log:
>> >
>> > Sep 3, 2016 7:31:07 PMSnapshot 'Auto-generated for Live Storage
>> > Migration' deletion for VM 'lv01' has been completed.
>> > Sep 3, 2016 6:46:46 PMSnapshot 'Auto-generated for Live Storage
>> > Migration' deletion for VM 'lv01' was initiated by SYSTEM
>> >
>> > And the related tasks
>> >
>> > Removing Snapshot Auto-generated for Live Storage Migration of VM lv01
>> > Sep 3, 2016 6:46:44 PMN/A29f45ca9
>> > ValidatingSep 3, 2016 6:46:44 PMuntilSep 3, 2016 6:46:44 PM
>> > ExecutingSep 3, 2016 6:46:44 PMuntilSep 3, 2016 7:31:06 PM
>> >
>> > FinalizingSep 3, 2016 7:31:06 PMN/A
>> >
>> >
>> >
>> > On 4 September 2016 at 14:27, Nir Soffer  wrote:
>> >>
>> >> On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett <
>> mat...@ltresources.co.uk>
>> >> wrote:
>> >>>
>> >>> How do I fix / kill a hung vdsm task?
>> >>>
>> >>> It seems to have completed the task but is stuck finalising.
>> >>>
>> >>> Removing Snapshot Auto-generated for Live Storage Migration
>> >>> Validating
>> >>> Executing
>> >>> (hour glass) Finalizing
>> >>>
>> >>> Task has been 'stuck' finalising for over 13 hours
>> >>
>> >>
>> >> Can you share engine and vdsm logs since the time the merge was
>> started?
>> >>
>> >> Nir
>> >
>> >
>>
>> ,
>>
>> ___
>> Users mailing list
>> Users@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>>
>>
>> --
>> С уважением, Краснобаев Михаил.
>>
>>
>>
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Hung task finalizing live migration

[Краснобаев Михаил]

Hi, There is a way to cancel a running task  -  look here http://lists.ovirt.org/pipermail/users/2014-November/028946.htmlI was able to stop snapshot deletion this way. Best, Mikhail. 08.09.2016, 08:14, "Maton, Brett" :Any suggestions ?THe task has been hung for 5 days now, I can't start the machine or destroy it.On 7 September 2016 at 06:49, Maton, Brett  wrote:Sorry just hit replyI'm seeing these errors in the logs which look related to the problem:2016-09-07 06:46:35,123 ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (DefaultQuartzScheduler6) [19c58c0d] Failed invoking callback end method 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with exception 'null', the callback is marked for end method retries2016-09-07 06:46:45,184 ERROR [org.ovirt.engine.core.bll.CommandsFactory] (DefaultQuartzScheduler7) [19c58c0d] Error in invocating CTOR of command 'LiveMigrateDisk': null2016-09-07 06:46:45,185 ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (DefaultQuartzScheduler7) [19c58c0d] Failed invoking callback end method 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with exception 'null', the callback is marked for end method retriesOn 5 September 2016 at 06:46, Nir Soffer  wrote:Hi Maton,  Please reply to the list, not to me directly.  Ala, can you look at this? is this a known issue?  Thanks, Nir On Mon, Sep 5, 2016 at 8:43 AM, Maton, Brett  wrote: > Log files as requested > > https://ufile.io/4fc35 vdsm log > https://ufile.io/e9836 engine 03-Sep > https://ufile.io/15f37 engine 04-Sep > > vdsm log stops on the 01-Sep... > > Couple of entries from the event log: > > Sep 3, 2016 7:31:07 PM    Snapshot 'Auto-generated for Live Storage > Migration' deletion for VM 'lv01' has been completed. > Sep 3, 2016 6:46:46 PM    Snapshot 'Auto-generated for Live Storage > Migration' deletion for VM 'lv01' was initiated by SYSTEM > > And the related tasks > > Removing Snapshot Auto-generated for Live Storage Migration of VM lv01 > Sep 3, 2016 6:46:44 PM        N/A    29f45ca9 > Validating    Sep 3, 2016 6:46:44 PM    until    Sep 3, 2016 6:46:44 PM > Executing    Sep 3, 2016 6:46:44 PM    until    Sep 3, 2016 7:31:06 PM > > Finalizing    Sep 3, 2016 7:31:06 PM        N/A > > > > On 4 September 2016 at 14:27, Nir Soffer  wrote: >> >> On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett  >> wrote: >>> >>> How do I fix / kill a hung vdsm task? >>> >>> It seems to have completed the task but is stuck finalising. >>> >>> Removing Snapshot Auto-generated for Live Storage Migration >>> Validating >>> Executing >>> (hour glass) Finalizing >>> >>> Task has been 'stuck' finalising for over 13 hours >> >> >> Can you share engine and vdsm logs since the time the merge was started? >> >> Nir > >,___Users mailing listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users  -- С уважением, Краснобаев Михаил.  ___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Hung task finalizing live migration

[Maton, Brett]

Thanks for the pointer Mikhail, however I don't get any tasks listed with
that command:

vdsClient -s 0 getAllTasksStatuses

/usr/share/vdsm/vdsClient.py:33: DeprecationWarning: vdscli uses xmlrpc.
since ovirt 3.6 xmlrpc is deprecated, please use vdsm.jsonrpcvdscli
  from vdsm import utils, vdscli, constants

{'status': {'message': 'OK', 'code': 0}, 'allTasksStatus': {}}


On 8 September 2016 at 09:51, Краснобаев Михаил  wrote:

> Hi,
>
> There is a way to cancel a running task  -  look here
> http://lists.ovirt.org/pipermail/users/2014-November/028946.html
> I was able to stop snapshot deletion this way.
>
> Best, Mikhail.
>
> 08.09.2016, 08:14, "Maton, Brett" :
>
> Any suggestions ?
>
> THe task has been hung for 5 days now, I can't start the machine or
> destroy it.
>
>
> On 7 September 2016 at 06:49, Maton, Brett 
> wrote:
>
> Sorry just hit reply
>
> I'm seeing these errors in the logs which look related to the problem:
>
>
> 2016-09-07 06:46:35,123 ERROR 
> [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller]
> (DefaultQuartzScheduler6) [19c58c0d] Failed invoking callback end method
> 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with
> exception 'null', the callback is marked for end method retries
> 2016-09-07 06:46:45,184 ERROR [org.ovirt.engine.core.bll.CommandsFactory]
> (DefaultQuartzScheduler7) [19c58c0d] Error in invocating CTOR of command
> 'LiveMigrateDisk': null
> 2016-09-07 06:46:45,185 ERROR 
> [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller]
> (DefaultQuartzScheduler7) [19c58c0d] Failed invoking callback end method
> 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with
> exception 'null', the callback is marked for end method retries
>
> On 5 September 2016 at 06:46, Nir Soffer  wrote:
>
> Hi Maton,
>
> Please reply to the list, not to me directly.
>
> Ala, can you look at this? is this a known issue?
>
> Thanks,
> Nir
>
> On Mon, Sep 5, 2016 at 8:43 AM, Maton, Brett 
> wrote:
> > Log files as requested
> >
> > https://ufile.io/4fc35 vdsm log
> > https://ufile.io/e9836 engine 03-Sep
> > https://ufile.io/15f37 engine 04-Sep
> >
> > vdsm log stops on the 01-Sep...
> >
> > Couple of entries from the event log:
> >
> > Sep 3, 2016 7:31:07 PMSnapshot 'Auto-generated for Live Storage
> > Migration' deletion for VM 'lv01' has been completed.
> > Sep 3, 2016 6:46:46 PMSnapshot 'Auto-generated for Live Storage
> > Migration' deletion for VM 'lv01' was initiated by SYSTEM
> >
> > And the related tasks
> >
> > Removing Snapshot Auto-generated for Live Storage Migration of VM lv01
> > Sep 3, 2016 6:46:44 PMN/A29f45ca9
> > ValidatingSep 3, 2016 6:46:44 PMuntilSep 3, 2016 6:46:44 PM
> > ExecutingSep 3, 2016 6:46:44 PMuntilSep 3, 2016 7:31:06 PM
> >
> > FinalizingSep 3, 2016 7:31:06 PMN/A
> >
> >
> >
> > On 4 September 2016 at 14:27, Nir Soffer  wrote:
> >>
> >> On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett  >
> >> wrote:
> >>>
> >>> How do I fix / kill a hung vdsm task?
> >>>
> >>> It seems to have completed the task but is stuck finalising.
> >>>
> >>> Removing Snapshot Auto-generated for Live Storage Migration
> >>> Validating
> >>> Executing
> >>> (hour glass) Finalizing
> >>>
> >>> Task has been 'stuck' finalising for over 13 hours
> >>
> >>
> >> Can you share engine and vdsm logs since the time the merge was started?
> >>
> >> Nir
> >
> >
>
> ,
>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
>
> --
> С уважением, Краснобаев Михаил.
>
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Hung task finalizing live migration

[Maton, Brett]

Any suggestions ?

THe task has been hung for 5 days now, I can't start the machine or destroy
it.



On 7 September 2016 at 06:49, Maton, Brett  wrote:

> Sorry just hit reply
>
> I'm seeing these errors in the logs which look related to the problem:
>
>
> 2016-09-07 06:46:35,123 ERROR 
> [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller]
> (DefaultQuartzScheduler6) [19c58c0d] Failed invoking callback end method
> 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with
> exception 'null', the callback is marked for end method retries
> 2016-09-07 06:46:45,184 ERROR [org.ovirt.engine.core.bll.CommandsFactory]
> (DefaultQuartzScheduler7) [19c58c0d] Error in invocating CTOR of command
> 'LiveMigrateDisk': null
> 2016-09-07 06:46:45,185 ERROR 
> [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller]
> (DefaultQuartzScheduler7) [19c58c0d] Failed invoking callback end method
> 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with
> exception 'null', the callback is marked for end method retries
>
> On 5 September 2016 at 06:46, Nir Soffer  wrote:
>
>> Hi Maton,
>>
>> Please reply to the list, not to me directly.
>>
>> Ala, can you look at this? is this a known issue?
>>
>> Thanks,
>> Nir
>>
>> On Mon, Sep 5, 2016 at 8:43 AM, Maton, Brett 
>> wrote:
>> > Log files as requested
>> >
>> > https://ufile.io/4fc35 vdsm log
>> > https://ufile.io/e9836 engine 03-Sep
>> > https://ufile.io/15f37 engine 04-Sep
>> >
>> > vdsm log stops on the 01-Sep...
>> >
>> > Couple of entries from the event log:
>> >
>> > Sep 3, 2016 7:31:07 PMSnapshot 'Auto-generated for Live Storage
>> > Migration' deletion for VM 'lv01' has been completed.
>> > Sep 3, 2016 6:46:46 PMSnapshot 'Auto-generated for Live Storage
>> > Migration' deletion for VM 'lv01' was initiated by SYSTEM
>> >
>> > And the related tasks
>> >
>> > Removing Snapshot Auto-generated for Live Storage Migration of VM lv01
>> > Sep 3, 2016 6:46:44 PMN/A29f45ca9
>> > ValidatingSep 3, 2016 6:46:44 PMuntilSep 3, 2016 6:46:44 PM
>> > ExecutingSep 3, 2016 6:46:44 PMuntilSep 3, 2016 7:31:06 PM
>> >
>> > FinalizingSep 3, 2016 7:31:06 PMN/A
>> >
>> >
>> >
>> > On 4 September 2016 at 14:27, Nir Soffer  wrote:
>> >>
>> >> On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett <
>> mat...@ltresources.co.uk>
>> >> wrote:
>> >>>
>> >>> How do I fix / kill a hung vdsm task?
>> >>>
>> >>> It seems to have completed the task but is stuck finalising.
>> >>>
>> >>> Removing Snapshot Auto-generated for Live Storage Migration
>> >>> Validating
>> >>> Executing
>> >>> (hour glass) Finalizing
>> >>>
>> >>> Task has been 'stuck' finalising for over 13 hours
>> >>
>> >>
>> >> Can you share engine and vdsm logs since the time the merge was
>> started?
>> >>
>> >> Nir
>> >
>> >
>>
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Hung task finalizing live migration

[Maton, Brett]

Sorry just hit reply

I'm seeing these errors in the logs which look related to the problem:


2016-09-07 06:46:35,123 ERROR
[org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller]
(DefaultQuartzScheduler6) [19c58c0d] Failed invoking callback end method
'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with
exception 'null', the callback is marked for end method retries
2016-09-07 06:46:45,184 ERROR [org.ovirt.engine.core.bll.CommandsFactory]
(DefaultQuartzScheduler7) [19c58c0d] Error in invocating CTOR of command
'LiveMigrateDisk': null
2016-09-07 06:46:45,185 ERROR
[org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller]
(DefaultQuartzScheduler7) [19c58c0d] Failed invoking callback end method
'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with
exception 'null', the callback is marked for end method retries

On 5 September 2016 at 06:46, Nir Soffer  wrote:

> Hi Maton,
>
> Please reply to the list, not to me directly.
>
> Ala, can you look at this? is this a known issue?
>
> Thanks,
> Nir
>
> On Mon, Sep 5, 2016 at 8:43 AM, Maton, Brett 
> wrote:
> > Log files as requested
> >
> > https://ufile.io/4fc35 vdsm log
> > https://ufile.io/e9836 engine 03-Sep
> > https://ufile.io/15f37 engine 04-Sep
> >
> > vdsm log stops on the 01-Sep...
> >
> > Couple of entries from the event log:
> >
> > Sep 3, 2016 7:31:07 PMSnapshot 'Auto-generated for Live Storage
> > Migration' deletion for VM 'lv01' has been completed.
> > Sep 3, 2016 6:46:46 PMSnapshot 'Auto-generated for Live Storage
> > Migration' deletion for VM 'lv01' was initiated by SYSTEM
> >
> > And the related tasks
> >
> > Removing Snapshot Auto-generated for Live Storage Migration of VM lv01
> > Sep 3, 2016 6:46:44 PMN/A29f45ca9
> > ValidatingSep 3, 2016 6:46:44 PMuntilSep 3, 2016 6:46:44 PM
> > ExecutingSep 3, 2016 6:46:44 PMuntilSep 3, 2016 7:31:06 PM
> >
> > FinalizingSep 3, 2016 7:31:06 PMN/A
> >
> >
> >
> > On 4 September 2016 at 14:27, Nir Soffer  wrote:
> >>
> >> On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett  >
> >> wrote:
> >>>
> >>> How do I fix / kill a hung vdsm task?
> >>>
> >>> It seems to have completed the task but is stuck finalising.
> >>>
> >>> Removing Snapshot Auto-generated for Live Storage Migration
> >>> Validating
> >>> Executing
> >>> (hour glass) Finalizing
> >>>
> >>> Task has been 'stuck' finalising for over 13 hours
> >>
> >>
> >> Can you share engine and vdsm logs since the time the merge was started?
> >>
> >> Nir
> >
> >
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Hung task finalizing live migration

[Nir Soffer]

On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett 
wrote:

> How do I fix / kill a hung vdsm task?
>
> It seems to have completed the task but is stuck finalising.
>
> Removing Snapshot Auto-generated for Live Storage Migration
> Validating
> Executing
> (hour glass) Finalizing
>
> Task has been 'stuck' finalising for over 13 hours
>

Can you share engine and vdsm logs since the time the merge was started?

Nir
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users