Re: [ovirt-users] how to clean stuck task

[Gianluca Cecchi]

In the mean time, as I had to give an answer for the snapshotted VM, I
decided to follow one of the suggestions to run engine-setup and so also to
pass my engine from 4.1.6 to 4.1.7.
And indeed the 2 stale tasks have been cleaned.
The lock symbol has gone away from the apex VM too.

Probably the steps solving the problems were these during engine-setup:

[ INFO  ] Cleaning async tasks and compensations
[ INFO  ] Unlocking existing entities

Does this mean that in general I can also run engine-setup without
upgrading at all? Is the clean part run also in that case or only during
effective upgrades?

I initiated a clone of the taken snapshot on apex VM and it seems to go
correctly and in task pane I see only that task and no more.

In SPM now I have indeed

[root@ov300 ~]# vdsClient -s 0 getAllTasksStatuses
{'status': {'message': 'OK', 'code': 0}, 'allTasksStatus':
{'20fa401f-b6f8-43f5-b0fd-6767d46e2335': {'message': 'running job 1 of 1',
'code': 0, 'taskID': '20fa401f-b6f8-43f5-b0fd-6767d46e2335', 'taskResult':
'', 'taskState': 'running'}}}

[root@ov300 ~]#

It should take about half an hour to complete and I will see.

Anyway in my opinion it would be nice to have some more in deep
documentation about how to run taskcleaner.sh or simply officially say to
leave it to developers if this is the intended case (or to Red Hat support
in case of RHEV usage)

Cheers,
Gianluca
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] how to clean stuck task

[Gianluca Cecchi]

On Mon, Nov 13, 2017 at 8:53 PM, Fred Rolland  wrote:

> For cleaning these "Tasks" entries (which are actually Jobs in the DB),
> you will need to use taskcleaner.sh.
> The unlock_entity.sh is for removing locks on entities.
>

ok. Thanks for clarifying the concept.
But from the inline help of taskcleaner.sh it is not so clear to me which
option makes it display vs remove...
Perhaps without options it only display? or with "-T" it first displays
asking if I want to remove "all comands that have running tasks"?
How can I only display?


> Before running the task cleaner, you need to be sure you actually don't
> have any tasks running in the SPM.
>

How can I check this?
Is still valid this command I found on a previous (around 2014) thread and
executed on the SPM (as shown in web admin portal) host right now?

[root@ov300 vdsm]# vdsClient -s 0 getAllTasksStatuses
{'status': {'message': 'OK', 'code': 0}, 'allTasksStatus': {}}

[root@ov300 vdsm]#

Does it mean no task?



> Regarding the snapshot creation on 'apex", can you provide the logs around
> the time of the operation?
>

engine.log is here:
https://drive.google.com/file/d/1nD0p90ZvwphrEafgKZkCCquNu_
vVY8p8/view?usp=sharing

vdsm log of the SPM host in .xz format here:
https://drive.google.com/file/d/1Zzk04W6400wPh1sM8O4WyFDAGQaqa
UZK/view?usp=sharing

Thanks,
Gianluca
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] how to clean stuck task

[Fred Rolland]

For cleaning these "Tasks" entries (which are actually Jobs in the DB), you
will need to use taskcleaner.sh.
The unlock_entity.sh is for removing locks on entities.

Before running the task cleaner, you need to be sure you actually don't
have any tasks running in the SPM.

Regarding the snapshot creation on 'apex", can you provide the logs around
the time of the operation?



On Mon, Nov 13, 2017 at 8:06 PM, Gianluca Cecchi 
wrote:

> On Mon, Nov 13, 2017 at 7:01 PM, Gianluca Cecchi <
> gianluca.cec...@gmail.com> wrote:
>
>>
>>
>>
>> I'd prefer to avoid direct manage of the rdbms...
>>
>> As it happened on Friday, after running again the command
>>
>> PGPASSWORD=... /usr/share/ovirt-engine/setup/dbutils/unlock_entity.sh -t
>> all -u engine
>>
>> now the gui after some time shows no active task
>>
>> but the vm still shows as locked and in the event pane I only have the
>> message
>> Snapshot '13-11-2017-16-30-Danilo' creation for VM 'apex' was initiated
>> by user1@internal-authz.
>> but no message about its correct completion
>>
>> This is my disks' subtab situation for the vm, where it seems all ok:
>> https://drive.google.com/file/d/1tF3VdA8gzMVvtn2x4HTKOmltRmU
>> R7cTG/view?usp=sharing
>>
>> And this the snapshots' subtab where it seems all ok:
>> https://drive.google.com/file/d/1qbEhdS6eOZL-f8y5xCnuc_4_HsY
>> M4mAa/view?usp=sharing
>>
>> But the VM has the lock symbol in side of itself.
>> Any guidance appreciated, both to solve the problem and to understand if
>> the snapshot is consistent, as a user should to changes to the VM with a
>> chance to have to come back
>>
>> Gianluca
>>
>
> I have just verified that if I run a new task (in my case has been the
> removal of the snaptest VM), suddenly I see from 0 to 3 tasks, with the
> deletion task completing with success, while the 2 stale tasks appearing
> again in the task pane where they disappeared
> see here:
> https://drive.google.com/file/d/1xcbJsI02iSEAHYEiFYkRWvz9SW_
> ElsIY/view?usp=sharing
>
>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] how to clean stuck task

[Gianluca Cecchi]

On Mon, Nov 13, 2017 at 7:01 PM, Gianluca Cecchi 
wrote:

>
>
>
> I'd prefer to avoid direct manage of the rdbms...
>
> As it happened on Friday, after running again the command
>
> PGPASSWORD=... /usr/share/ovirt-engine/setup/dbutils/unlock_entity.sh -t
> all -u engine
>
> now the gui after some time shows no active task
>
> but the vm still shows as locked and in the event pane I only have the
> message
> Snapshot '13-11-2017-16-30-Danilo' creation for VM 'apex' was initiated by
> user1@internal-authz.
> but no message about its correct completion
>
> This is my disks' subtab situation for the vm, where it seems all ok:
> https://drive.google.com/file/d/1tF3VdA8gzMVvtn2x4HTKOmltRmUR7
> cTG/view?usp=sharing
>
> And this the snapshots' subtab where it seems all ok:
> https://drive.google.com/file/d/1qbEhdS6eOZL-f8y5xCnuc_4_
> HsYM4mAa/view?usp=sharing
>
> But the VM has the lock symbol in side of itself.
> Any guidance appreciated, both to solve the problem and to understand if
> the snapshot is consistent, as a user should to changes to the VM with a
> chance to have to come back
>
> Gianluca
>

I have just verified that if I run a new task (in my case has been the
removal of the snaptest VM), suddenly I see from 0 to 3 tasks, with the
deletion task completing with success, while the 2 stale tasks appearing
again in the task pane where they disappeared
see here:
https://drive.google.com/file/d/1xcbJsI02iSEAHYEiFYkRWvz9SW_ElsIY/view?usp=sharing
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] how to clean stuck task

[Gianluca Cecchi]

On Mon, Nov 13, 2017 at 5:28 PM, Wesley Stewart  wrote:

> Is the VM showing as locked?  You could try to manually unlocking it.  I
> did this in a test environment a while back with some success after a vm
> was stuck in a "locked" state.  Of course the gui and engine should handle
> most of this for you, manually mucking around the internal DB can cause
> some pretty serious issues if you are not careful...
>
> sudo su postgres
>
> psql -d engine -U postgres
>
> SELECT vm_guid, vm_name FROM vm_static WHERE vm_name='*VM_Name_Here*';
> This should return a string such as: "0ec20854-e1ca-4e49-be87-
> a6cd36d40c18"
>
> Reset the lock:
> update vm_dynamic SET status=0 where vm_guid='0ec20854-e1ca-4e49-
> be87-a6cd36d40c18';
>
>
>
>
I'd prefer to avoid direct manage of the rdbms...

As it happened on Friday, after running again the command

PGPASSWORD=... /usr/share/ovirt-engine/setup/dbutils/unlock_entity.sh -t
all -u engine

now the gui after some time shows no active task

but the vm still shows as locked and in the event pane I only have the
message
Snapshot '13-11-2017-16-30-Danilo' creation for VM 'apex' was initiated by
user1@internal-authz.
but no message about its correct completion

This is my disks' subtab situation for the vm, where it seems all ok:
https://drive.google.com/file/d/1tF3VdA8gzMVvtn2x4HTKOmltRmUR7cTG/view?usp=sharing

And this the snapshots' subtab where it seems all ok:
https://drive.google.com/file/d/1qbEhdS6eOZL-f8y5xCnuc_4_HsYM4mAa/view?usp=sharing

But the VM has the lock symbol in side of itself.
Any guidance appreciated, both to solve the problem and to understand if
the snapshot is consistent, as a user should to changes to the VM with a
chance to have to come back

Gianluca
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] how to clean stuck task

[Wesley Stewart]

Is the VM showing as locked?  You could try to manually unlocking it.  I
did this in a test environment a while back with some success after a vm
was stuck in a "locked" state.  Of course the gui and engine should handle
most of this for you, manually mucking around the internal DB can cause
some pretty serious issues if you are not careful...

sudo su postgres

psql -d engine -U postgres

SELECT vm_guid, vm_name FROM vm_static WHERE vm_name='*VM_Name_Here*';
This should return a string such as: "0ec20854-e1ca-4e49-be87-a6cd36d40c18"

Reset the lock:
update vm_dynamic SET status=0 where vm_guid='
0ec20854-e1ca-4e49-be87-a6cd36d40c18';



On Mon, Nov 13, 2017 at 11:01 AM, Gianluca Cecchi  wrote:

> Hello,
>
> it seems the same lock (with the same date) has come again in the list of
> tasks.
> Now I have two things in "supposed running" tasks:
> - the "Restoring VM Snapshot Active VM before the preview of VM snaptest"
> of 8/11
> - the "Creating VM Snapshot 13-11-2017-16-30-Danilo for VM apex"  (now
> running since half an hour for a vm that has only one 90Gb disk on SAN FC
> storage domain)
>
> It was not me to run the "create snapshot" command today, so I'm not sure
> if it was the snapshot creation job responsible of letting come out again
> the former hung task... or if it was already there before...
> Any further hint?
> How can I manually verify the status of the snapshot creation task?
>
> Gianluca
>
>
> On Fri, Nov 10, 2017 at 5:01 PM, Gianluca Cecchi <
> gianluca.cec...@gmail.com> wrote:
>
>> On Fri, Nov 10, 2017 at 3:48 PM,  wrote:
>>
>>>

>>> I've seen this behavior too. IIRC the stale cleaning was not instant, it
>>> took some time to be applied.
>>>
>>> Regards.
>>>
>>> Gianluca

>>>
>> Confirmed.
>> Quite soon after the command I saw that the status of the "Current"
>> snapshot line changed from Locked (it was so since 8/11) to OK, but the
>> task remained at least for half an hour.
>> Now, after about one hour and a half I connected again to the web admin
>> gui and I see 0 Tasks, so the problem has been resolved.
>>
>> Thanks again,
>> Gianluca
>>
>>
>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] how to clean stuck task

[Gianluca Cecchi]

Hello,

it seems the same lock (with the same date) has come again in the list of
tasks.
Now I have two things in "supposed running" tasks:
- the "Restoring VM Snapshot Active VM before the preview of VM snaptest"
of 8/11
- the "Creating VM Snapshot 13-11-2017-16-30-Danilo for VM apex"  (now
running since half an hour for a vm that has only one 90Gb disk on SAN FC
storage domain)

It was not me to run the "create snapshot" command today, so I'm not sure
if it was the snapshot creation job responsible of letting come out again
the former hung task... or if it was already there before...
Any further hint?
How can I manually verify the status of the snapshot creation task?

Gianluca


On Fri, Nov 10, 2017 at 5:01 PM, Gianluca Cecchi 
wrote:

> On Fri, Nov 10, 2017 at 3:48 PM,  wrote:
>
>>
>>>
>> I've seen this behavior too. IIRC the stale cleaning was not instant, it
>> took some time to be applied.
>>
>> Regards.
>>
>> Gianluca
>>>
>>
> Confirmed.
> Quite soon after the command I saw that the status of the "Current"
> snapshot line changed from Locked (it was so since 8/11) to OK, but the
> task remained at least for half an hour.
> Now, after about one hour and a half I connected again to the web admin
> gui and I see 0 Tasks, so the problem has been resolved.
>
> Thanks again,
> Gianluca
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] how to clean stuck task

[Gianluca Cecchi]

On Fri, Nov 10, 2017 at 3:48 PM,  wrote:

>
>>
> I've seen this behavior too. IIRC the stale cleaning was not instant, it
> took some time to be applied.
>
> Regards.
>
> Gianluca
>>
>
Confirmed.
Quite soon after the command I saw that the status of the "Current"
snapshot line changed from Locked (it was so since 8/11) to OK, but the
task remained at least for half an hour.
Now, after about one hour and a half I connected again to the web admin gui
and I see 0 Tasks, so the problem has been resolved.

Thanks again,
Gianluca
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] how to clean stuck task

[Wesley Stewart]

You could also go database diving.  I had an issue where I tried to import
a VM from my export domain and it just got hung.  I tried running the
unlock_entity script but it just kept failing.  It sat there for months
stuck, and found
http://lists.ovirt.org/pipermail/users/2015-April/032346.html

Of course deleting something from your database is quite permanent. I would
wait and upgrade to 4.1.7, but something like the below should work. But
probably not recommended

Drop into postgres
psql -d engine -U postgres

List your tasks and grab the job_id
select * from job order by start_time desc;

select DeleteJob('8424f7a9-2a4c-4567-b528-45bbc1c2534f');
Where the string here is the job ID

On Fri, Nov 10, 2017 at 9:48 AM,  wrote:

> El 2017-11-10 14:41, Gianluca Cecchi escribió:
>
>> On Fri, Nov 10, 2017 at 3:34 PM,  wrote:
>>
>> oVirt upgrade to 4.1.7 will probably cleanup this stale task.
>>> However, if you want to do it before upgrading, run this command:
>>>
>>>PGPASSWORD=...
>>> /usr/share/ovirt-engine/setup/dbutils/unlock_entity.sh -t all -u
>>> engine
>>>
>>> Note that unlock_entity.sh has many flags and this is just an
>>> example (should clean all stale tasks).
>>>
>>> You can find the PGPASSWORD value in the
>>> /etc/ovirt-engine/engine.conf.d/10-setup-database.conf file. As of
>>> 4.2 you won't need to supply credentials anymore [1].
>>>
>>> Regards,
>>>
>>> Nicolás
>>>
>>
>> It seems it didn't work as expected.
>> I got this at command line output
>>
>> "
>>
>> select fn_db_unlock_all();
>>
>>
>> INSERT 0 1
>> unlock all  completed successfully.
>> "
>>
>>
> This is expected.
>
> But the task remains in webadmin gui and I got an alert message in
>> alert section, of this type
>> "
>> /usr/share/ovirt-engine/setup/dbutils/unlock_entity.sh : System user
>> root run manually unlock_entity script on entity [type,id] [all,] with
>> db user engine
>> "
>>
>>
> I've seen this behavior too. IIRC the stale cleaning was not instant, it
> took some time to be applied.
>
> Regards.
>
> Gianluca
>>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] how to clean stuck task

[nicolas]

El 2017-11-10 14:41, Gianluca Cecchi escribió:

On Fri, Nov 10, 2017 at 3:34 PM,  wrote:


oVirt upgrade to 4.1.7 will probably cleanup this stale task.
However, if you want to do it before upgrading, run this command:

   PGPASSWORD=...
/usr/share/ovirt-engine/setup/dbutils/unlock_entity.sh -t all -u
engine

Note that unlock_entity.sh has many flags and this is just an
example (should clean all stale tasks).

You can find the PGPASSWORD value in the
/etc/ovirt-engine/engine.conf.d/10-setup-database.conf file. As of
4.2 you won't need to supply credentials anymore [1].

Regards,

Nicolás


It seems it didn't work as expected.
I got this at command line output

"

select fn_db_unlock_all();
 

INSERT 0 1
unlock all  completed successfully.
"



This is expected.


But the task remains in webadmin gui and I got an alert message in
alert section, of this type
"
/usr/share/ovirt-engine/setup/dbutils/unlock_entity.sh : System user
root run manually unlock_entity script on entity [type,id] [all,] with
db user engine
"



I've seen this behavior too. IIRC the stale cleaning was not instant, it 
took some time to be applied.


Regards.


Gianluca

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] how to clean stuck task

[Gianluca Cecchi]

On Fri, Nov 10, 2017 at 3:34 PM,  wrote:

> oVirt upgrade to 4.1.7 will probably cleanup this stale task. However, if
> you want to do it before upgrading, run this command:
>
>PGPASSWORD=... /usr/share/ovirt-engine/setup/dbutils/unlock_entity.sh
> -t all -u engine
>
> Note that unlock_entity.sh has many flags and this is just an example
> (should clean all stale tasks).
>
> You can find the PGPASSWORD value in the 
> /etc/ovirt-engine/engine.conf.d/10-setup-database.conf
> file. As of 4.2 you won't need to supply credentials anymore [1].
>
> Regards,
>
> Nicolás
>


It seems it didn't work as expected.
I got this at command line output

"
select fn_db_unlock_all();


INSERT 0 1
unlock all  completed successfully.
"

But the task remains in webadmin gui and I got an alert message in alert
section, of this type
"
/usr/share/ovirt-engine/setup/dbutils/unlock_entity.sh : System user root
run manually unlock_entity script on entity [type,id] [all,] with db user
engine
"

Gianluca
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] how to clean stuck task

[nicolas]

oVirt upgrade to 4.1.7 will probably cleanup this stale task. However, 
if you want to do it before upgrading, run this command:


   PGPASSWORD=... /usr/share/ovirt-engine/setup/dbutils/unlock_entity.sh 
-t all -u engine


Note that unlock_entity.sh has many flags and this is just an example 
(should clean all stale tasks).


You can find the PGPASSWORD value in the 
/etc/ovirt-engine/engine.conf.d/10-setup-database.conf file. As of 4.2 
you won't need to supply credentials anymore [1].


Regards,

Nicolás

  [1]: https://gerrit.ovirt.org/82615

El 2017-11-10 14:16, Gianluca Cecchi escribió:

Hello, 
I have a task that seems stuck in webadmin gui, in the sens tha I have
"Tasks(1)" listed
The task is  Restoring VM Snapshot Active VM before the preview of
VM snaptest
and the VM is powered down.
Screenshot of expanded steps of task, that actually seem all
completed, is here:
https://drive.google.com/file/d/1bfl_gEfVotIrxGC9TDzPHPCeRub41mUa/view?usp=sharing
[1]

Any hint on what to do to clean things? I'm on oVirt
4.1.6.2-1.el7.centos and I would like to clean before upgrading to
4.1.7.

Thanks
Gianluca

Links:
--
[1]
https://drive.google.com/file/d/1bfl_gEfVotIrxGC9TDzPHPCeRub41mUa/view?usp=sharing

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users