-----Original Message----- From: Jochim, Ingo [mailto:[email protected]] Sent: jeudi 25 juin 2015 08:15 To: '[email protected]' Subject: AW: ACS does shutdown some VMs
Hello all, any idea how to cleanup the job table? Thanks and regards, Ingo -----Ursprüngliche Nachricht----- Von: Jochim, Ingo [mailto:[email protected]] Gesendet: Dienstag, 23. Juni 2015 14:59 An: '[email protected]' Betreff: AW: ACS does shutdown some VMs Hello Remi, ACS has version 4.4.1. This is the worker output. We never configured HA. How can we get rid of this? +--------+-----------+----------+---------------------+---------------------+-----------+------------+------------+ | VM | Type | VM state | Created (-2H) | HAworker step taken | Step | Hypervisor | Mgt server | +--------+-----------+----------+---------------------+---------------------+-----------+------------+------------+ | hts001 | Migration | Running | 2015-06-22 14:40:23 | None | Scheduled | hv01 | None | +--------+-----------+----------+---------------------+---------------------+-----------+------------+------------+ Note: Found 1 HA workers. The job queue is also quite long. What happens there? Regards, Ingo +----------+--------------+---------------+-----------+----------------------------------------------------------------------+-----------------------+---------------------+------------+--------+----------------+ | username | account_name | instance_name | vm_state | job_cmd | job_dispatcher | Job created | Mgt Server | Job ID | Related job ID | +----------+--------------+---------------+-----------+----------------------------------------------------------------------+-----------------------+---------------------+------------+--------+----------------+ | user01 | admin | None | None | com.cloud.vm.VmWorkReboot | VmWorkJobDispatcher | 2015-06-22 11:04:30 | None | 3791 | 3790 | | user01 | admin | None | None | com.cloud.vm.VmWorkRemoveNicFromVm | VmWorkJobDispatcher | 2015-06-23 10:30:01 | None | 3860 | 3859 | | user01 | admin | None | None | com.cloud.vm.VmWorkStart | VmWorkJobDispatcher | 2015-06-22 12:26:21 | None | 3817 | 3816 | | user01 | admin | None | None | com.cloud.vm.VmWorkStart | VmWorkJobDispatcher | 2015-06-22 13:38:06 | None | 3832 | 3831 | | user01 | admin | None | None | com.cloud.vm.VmWorkStart | VmWorkJobDispatcher | 2015-06-23 06:55:58 | None | 3857 | 3856 | | user01 | admin | None | None | com.cloud.vm.VmWorkStop | VmWorkJobDispatcher | 2015-06-22 09:21:34 | None | 3778 | 3777 | | user01 | admin | None | None | com.cloud.vm.VmWorkStop | VmWorkJobDispatcher | 2015-06-22 10:55:04 | None | 3789 | 3788 | | user01 | admin | None | None | org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-23 10:30:01 | None | 3859 | | | system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-18 08:01:43 | None | 3601 | 3590 | | system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-18 08:01:45 | None | 3602 | 3567 | | system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-18 08:01:47 | None | 3604 | 3559 | | system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-18 08:03:07 | None | 3616 | 3612 | | system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 11:53:36 | None | 3704 | 3609 | | system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 12:34:21 | None | 3718 | 3610 | | system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 12:34:21 | None | 3722 | 3611 | | system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 12:34:49 | None | 3733 | 3609 | | system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 13:01:36 | None | 3743 | 3610 | | system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 13:01:44 | None | 3749 | 3606 | | system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 12:08:26 | None | 3796 | 3698 | | system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 12:08:27 | None | 3798 | 3607 | | system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 12:09:32 | None | 3809 | 3615 | | system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 13:37:15 | None | 3825 | 3819 | | system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3836 | 3818 | | system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3837 | 3556 | | system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3839 | 3559 | | system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3841 | 3570 | | system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3842 | 3567 | | system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3845 | 3819 | | system | system | None | None | com.cloud.vm.VmWorkStart | VmWorkJobDispatcher | 2015-06-18 07:59:09 | None | 3600 | 3578 | | user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.RebootVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 11:04:30 | None | 3790 | | | user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 12:26:21 | None | 3816 | | | user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 13:38:06 | None | 3831 | | | user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 13:48:39 | None | 3835 | | | user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-23 06:55:58 | None | 3856 | | | user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StopVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 09:21:34 | None | 3777 | | | user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StopVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 10:55:04 | None | 3788 | | | system | system | s-32-VM | Expunging | None | pseudoJobDispatcher | 2015-06-20 11:53:33 | None | 3698 | | | system | system | s-33-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3609 | | | system | system | s-34-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3606 | | | system | system | s-35-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3611 | | | system | system | s-36-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3610 | | | system | system | s-37-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:03 | None | 3615 | | | system | system | s-38-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:03 | None | 3614 | | | system | system | s-39-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3612 | | | system | system | s-40-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3613 | | | system | system | s-41-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3607 | | | system | system | s-42-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3608 | | | system | system | s-43-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:47 | None | 3617 | | | system | system | s-44-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:47 | None | 3618 | | | system | system | s-45-VM | Expunging | None | pseudoJobDispatcher | 2015-06-20 12:35:04 | None | 3734 | | | system | system | s-82-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:52:09 | None | 3578 | | | system | system | s-83-VM | Expunging | None | pseudoJobDispatcher | 2015-06-22 13:37:12 | None | 3819 | | | system | system | s-84-VM | Expunging | None | pseudoJobDispatcher | 2015-06-22 13:37:12 | None | 3818 | | | system | system | s-85-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:52:09 | None | 3579 | | | system | system | s-86-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3559 | | | system | system | s-87-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:26 | None | 3570 | | | system | system | s-88-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3558 | | | system | system | s-89-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3560 | | | system | system | s-90-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3556 | | | system | system | s-91-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3557 | | | system | system | s-92-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:26 | None | 3567 | | | system | system | s-93-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:57:41 | None | 3590 | | | system | system | s-94-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3561 | | | system | system | s-95-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:26 | None | 3569 | | +----------+--------------+---------------+-----------+----------------------------------------------------------------------+-----------------------+---------------------+------------+--------+----------------+ Note: Found 64 running jobs. -----Ursprüngliche Nachricht----- Von: Remi Bergsma [mailto:[email protected]] Gesendet: Dienstag, 23. Juni 2015 09:41 An: [email protected] Betreff: Re: ACS does shutdown some VMs Hi, It’d double check the HA worker table to see if there are any left-over HA workers. Also, there could still be other jobs scheduled. If the API does not return anything useful, you may have a look at the DB itself. I wrote scripts to easily display HA workers [1] and jobs [2] which may help you as well. You can also execute the queries yourself, see [3] and [4] to get an idea. Regards, Remi [1] https://github.com/remibergsma/cloudstackOps/blob/master/listHAWorkers.py [2] https://github.com/remibergsma/cloudstackOps/blob/master/listRunningJobs.py [3] https://github.com/remibergsma/cloudstackOps/blob/master/cloudstackops/cloudstacksql.py#L76 [4] https://github.com/remibergsma/cloudstackOps/blob/master/cloudstackops/cloudstacksql.py#L109 On 23 Jun 2015, at 07:27, Tomasz Zięba <[email protected]<mailto:[email protected]>> wrote: What version of ACS ? Regards, Tomasz Zięba Twitter: @TZieba LinkedIn: pl.linkedin.com/pub/tomasz-zięba-ph-d/3b/7a8/ab6/<http://pl.linkedin.com/pub/tomasz-zięba-ph-d/3b/7a8/ab6/> <http://pl.linkedin.com/pub/tomasz-zi%C4%99ba-ph-d/3b/7a8/ab6/> sent from mobile 22 cze 2015 23:36 "Jochim, Ingo" <[email protected]<mailto:[email protected]>> napisał(a): Hello Andrei, The VM is not a HA VM. How can I check the queue? cloudmonkey list asyncjobs ??? isdefault = False # does this mean it should be off ? Regards, Ingo ________________________________________ Von: Andrei Mikhailovsky [[email protected]<mailto:[email protected]>] Gesendet: Montag, 22. Juni 2015 22:58 An: [email protected]<mailto:[email protected]> Betreff: Re: ACS does shutdown some VMs Hi Ingo, It might be an HA issue (check if these vms are set to HA). Also, it might be a broken job. Check out the jobs in the queue. Andrei ----- Original Message ----- From: "Ingo Jochim" <[email protected]<mailto:[email protected]>> To: [email protected]<mailto:[email protected]> Sent: Monday, 22 June, 2015 9:10:05 PM Subject: ACS does shutdown some VMs Hello all, we have a strange problem here. There are a couple of running VMs. As soon as ACS get's restarted it will start shutting down some of the VMs. It happens with the same VMs always. We can start those VMs without problems and they stay running. Is there some kind of destination state which might be wrong in the database? Thank you in advance. Ingo -- This email was Virus checked by UTM 9. http://www.sophos.com -- This email was Virus checked by UTM 9. http://www.sophos.com
