[ https://issues.apache.org/jira/browse/CLOUDSTACK-7564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Anthony Xu resolved CLOUDSTACK-7564. ------------------------------------ Resolution: Incomplete management server log SMlog, what's the zone(basic/advanced)? what's the network(w/o SG)? > [Automation][XenServer] Unable to Stop a VM - callHostPlugin failed for cmd: > destroy_network_rules_for_vm with args vmName: i-20-27-VM > -------------------------------------------------------------------------------------------------------------------------------------- > > Key: CLOUDSTACK-7564 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7564 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Automation, XenServer > Affects Versions: 4.5.0 > Reporter: Chandan Purushothama > Assignee: Anthony Xu > Priority: Blocker > Fix For: 4.5.0 > > > I see that the VM Stop Job failed due to the following reason: > *2014-09-16 15:51:21,914 WARN [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-76:ctx-abca3786) callHostPlugin failed for cmd: > destroy_network_rules_for_vm with args vmName: i-20-27-VM, due to There was > a failure communicating with the plugin. > 2014-09-16 15:51:21,915 DEBUG [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-76:ctx-abca3786) Catch exception > com.cloud.utils.exception.CloudRuntimeException when stop VM:i-20-27-VM due > to com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for > cmd: destroy_network_rules_for_vm with args vmName: i-20-27-VM, due to There > was a failure communicating with the plugin. > * > ======================== > VM Stop Job Logs Information: > ======================== > {noformat} > 2014-09-16 15:51:20,594 DEBUG [c.c.a.ApiServlet] > (catalina-exec-25:ctx-fedac54a) ===START=== 10.220.135.29 -- GET > jobid=3dc3e848-cf6f-4cb1-b05f-7d220bdf396a&apiKey=V9qdDxm-ufkQ7NG7IUBKZGbCo9gzC4d5pjKLwFqNDaLUDC3ELlMIGvqq6RjfF2EQ8qTC0GwfxbhswOFP-Hg-Cg&command=queryAsyncJobResult&response=json&signature=HFVB81DxD27cwGUnFn%2B2D3AQuRs%3D > 2014-09-16 15:51:20,597 DEBUG [c.c.a.ApiServlet] > (catalina-exec-22:ctx-8c7ba1c4) ===START=== 10.220.135.29 -- GET > jobid=a12f92f6-8efc-4518-b7b4-112cd1f40754&apiKey=V9qdDxm-ufkQ7NG7IUBKZGbCo9gzC4d5pjKLwFqNDaLUDC3ELlMIGvqq6RjfF2EQ8qTC0GwfxbhswOFP-Hg-Cg&command=queryAsyncJobResult&response=json&signature=%2F%2BLLeKMfokMZJ6pOg50PxPZSOjU%3D > 2014-09-16 15:51:20,629 DEBUG [c.c.u.AccountManagerImpl] > (API-Job-Executor-75:ctx-52918b50 job-208 ctx-22aa778f) Removed account 8 > 2014-09-16 15:51:20,633 DEBUG [c.c.a.ApiServlet] > (catalina-exec-22:ctx-8c7ba1c4 ctx-941f9ccb ctx-1af51dd9) ===END=== > 10.220.135.29 -- GET > jobid=a12f92f6-8efc-4518-b7b4-112cd1f40754&apiKey=V9qdDxm-ufkQ7NG7IUBKZGbCo9gzC4d5pjKLwFqNDaLUDC3ELlMIGvqq6RjfF2EQ8qTC0GwfxbhswOFP-Hg-Cg&command=queryAsyncJobResult&response=json&signature=%2F%2BLLeKMfokMZJ6pOg50PxPZSOjU%3D > 2014-09-16 15:51:20,648 DEBUG [c.c.a.ApiServlet] > (catalina-exec-25:ctx-fedac54a ctx-4056e031 ctx-353333e5) ===END=== > 10.220.135.29 -- GET > jobid=3dc3e848-cf6f-4cb1-b05f-7d220bdf396a&apiKey=V9qdDxm-ufkQ7NG7IUBKZGbCo9gzC4d5pjKLwFqNDaLUDC3ELlMIGvqq6RjfF2EQ8qTC0GwfxbhswOFP-Hg-Cg&command=queryAsyncJobResult&response=json&signature=HFVB81DxD27cwGUnFn%2B2D3AQuRs%3D > 2014-09-16 15:51:20,651 DEBUG [c.c.u.AccountManagerImpl] > (API-Job-Executor-75:ctx-52918b50 job-208 ctx-22aa778f) Successfully deleted > snapshots directories for all volumes under account 8 across all zones > 2014-09-16 15:51:20,655 DEBUG [c.c.u.AccountManagerImpl] > (API-Job-Executor-75:ctx-52918b50 job-208 ctx-22aa778f) Expunging # of vms > (accountId=8): 1 > 2014-09-16 15:51:20,655 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (API-Job-Executor-74:ctx-cbfc3d27 job-207 ctx-ea6e190d) Sync job-209 > execution on object VmWorkJobQueue.27 > 2014-09-16 15:51:20,658 WARN [c.c.u.d.Merovingian2] > (API-Job-Executor-74:ctx-cbfc3d27 job-207 ctx-ea6e190d) Was unable to find > lock for the key vm_instance27 and thread id 2057618920 > 2014-09-16 15:51:20,664 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (API-Job-Executor-75:ctx-52918b50 job-208 ctx-22aa778f) Sync job-210 > execution on object VmWorkJobQueue.7 > 2014-09-16 15:51:20,666 WARN [c.c.u.d.Merovingian2] > (API-Job-Executor-75:ctx-52918b50 job-208 ctx-22aa778f) Was unable to find > lock for the key vm_instance7 and thread id 1487507158 > 2014-09-16 15:51:20,928 DEBUG [c.c.a.m.AgentManagerImpl] > (AgentManager-Handler-15:null) SeqA 3-136: Processing Seq 3-136: { Cmd , > MgmtId: -1, via: 3, Ver: v1, Flags: 11, > [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":1,"_loadInfo":"{\n > \"connections\": []\n}","wait":0}}] } > 2014-09-16 15:51:20,932 DEBUG [c.c.a.m.AgentManagerImpl] > (AgentManager-Handler-15:null) SeqA 3-136: Sending Seq 3-136: { Ans: , > MgmtId: 125944753790399, via: 3, Ver: v1, Flags: 100010, > [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] } > 2014-09-16 15:51:21,586 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (AsyncJobMgr-Heartbeat-1:ctx-ee5979bc) Execute sync-queue item: > SyncQueueItemVO {id:62, queueId: 61, contentType: AsyncJob, contentId: 209, > lastProcessMsid: null, lastprocessNumber: null, lastProcessTime: null, > created: Tue Sep 16 15:51:20 UTC 2014} > 2014-09-16 15:51:21,587 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (AsyncJobMgr-Heartbeat-1:ctx-ee5979bc) Schedule queued job-209 > 2014-09-16 15:51:21,590 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (AsyncJobMgr-Heartbeat-1:ctx-ee5979bc) Execute sync-queue item: > SyncQueueItemVO {id:63, queueId: 6, contentType: AsyncJob, contentId: 210, > lastProcessMsid: null, lastprocessNumber: null, lastProcessTime: null, > created: Tue Sep 16 15:51:20 UTC 2014} > 2014-09-16 15:51:21,591 INFO [o.a.c.f.j.i.AsyncJobMonitor] > (Work-Job-Executor-62:ctx-7afe827c job-207/job-209) Add job-209 into job > monitoring > 2014-09-16 15:51:21,591 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (Work-Job-Executor-62:ctx-7afe827c job-207/job-209) Executing AsyncJobVO > {id:209, userId: 2, accountId: 2, instanceType: null, instanceId: null, cmd: > com.cloud.vm.VmWorkStop, cmdInfo: > rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAAG3QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwA, > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: > null, initMsid: 125944753790399, completeMsid: null, lastUpdated: null, > lastPolled: null, created: Tue Sep 16 15:51:20 UTC 2014} > 2014-09-16 15:51:21,591 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (AsyncJobMgr-Heartbeat-1:ctx-ee5979bc) Schedule queued job-210 > 2014-09-16 15:51:21,591 DEBUG [c.c.v.VmWorkJobDispatcher] > (Work-Job-Executor-62:ctx-7afe827c job-207/job-209) Run VM work job: > com.cloud.vm.VmWorkStop for VM 27, job origin: 207 > 2014-09-16 15:51:21,593 DEBUG [c.c.v.VmWorkJobHandlerProxy] > (Work-Job-Executor-62:ctx-7afe827c job-207/job-209 ctx-9abbad0d) Execute VM > work job: > com.cloud.vm.VmWorkStop{"cleanup":false,"userId":2,"accountId":2,"vmId":27,"handlerName":"VirtualMachineManagerImpl"} > 2014-09-16 15:51:21,597 INFO [o.a.c.f.j.i.AsyncJobMonitor] > (Work-Job-Executor-63:ctx-de88516b job-208/job-210) Add job-210 into job > monitoring > 2014-09-16 15:51:21,597 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (Work-Job-Executor-63:ctx-de88516b job-208/job-210) Executing AsyncJobVO > {id:210, userId: 2, accountId: 2, instanceType: null, instanceId: null, cmd: > com.cloud.vm.VmWorkStop, cmdInfo: > rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAAB3QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwA, > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: > null, initMsid: 125944753790399, completeMsid: null, lastUpdated: null, > lastPolled: null, created: Tue Sep 16 15:51:20 UTC 2014} > 2014-09-16 15:51:21,598 DEBUG [c.c.v.VmWorkJobDispatcher] > (Work-Job-Executor-63:ctx-de88516b job-208/job-210) Run VM work job: > com.cloud.vm.VmWorkStop for VM 7, job origin: 208 > 2014-09-16 15:51:21,599 DEBUG [c.c.v.VmWorkJobHandlerProxy] > (Work-Job-Executor-63:ctx-de88516b job-208/job-210 ctx-8dcff173) Execute VM > work job: > com.cloud.vm.VmWorkStop{"cleanup":false,"userId":2,"accountId":2,"vmId":7,"handlerName":"VirtualMachineManagerImpl"} > 2014-09-16 15:51:21,608 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-62:ctx-7afe827c job-207/job-209 ctx-9abbad0d) VM state > transitted from :Running to Stopping with event: StopRequestedvm's original > host id: 1 new host id: 1 host id before state transition: 1 > 2014-09-16 15:51:21,610 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-63:ctx-de88516b job-208/job-210 ctx-8dcff173) VM state > transitted from :Running to Stopping with event: StopRequestedvm's original > host id: 1 new host id: 1 host id before state transition: 1 > 2014-09-16 15:51:21,614 DEBUG [c.c.a.t.Request] > (Work-Job-Executor-62:ctx-7afe827c job-207/job-209 ctx-9abbad0d) Seq > 1-4070409638212796630: Sending { Cmd , MgmtId: 125944753790399, via: > 1(hpws460ca), Ver: v1, Flags: 100011, > [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"checkBeforeCleanup":false,"vmName":"i-20-27-VM","wait":0}}] > } > 2014-09-16 15:51:21,615 DEBUG [c.c.a.t.Request] > (Work-Job-Executor-62:ctx-7afe827c job-207/job-209 ctx-9abbad0d) Seq > 1-4070409638212796630: Executing: { Cmd , MgmtId: 125944753790399, via: > 1(hpws460ca), Ver: v1, Flags: 100011, > [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"checkBeforeCleanup":false,"vmName":"i-20-27-VM","wait":0}}] > } > 2014-09-16 15:51:21,615 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-76:ctx-abca3786) Seq 1-4070409638212796630: Executing request > 2014-09-16 15:51:21,616 DEBUG [c.c.a.t.Request] > (Work-Job-Executor-63:ctx-de88516b job-208/job-210 ctx-8dcff173) Seq > 1-4070409638212796631: Sending { Cmd , MgmtId: 125944753790399, via: > 1(hpws460ca), Ver: v1, Flags: 100011, > [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"checkBeforeCleanup":false,"vmName":"i-8-7-VM","wait":0}}] > } > 2014-09-16 15:51:21,616 DEBUG [c.c.a.t.Request] > (Work-Job-Executor-63:ctx-de88516b job-208/job-210 ctx-8dcff173) Seq > 1-4070409638212796631: Executing: { Cmd , MgmtId: 125944753790399, via: > 1(hpws460ca), Ver: v1, Flags: 100011, > [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"checkBeforeCleanup":false,"vmName":"i-8-7-VM","wait":0}}] > } > 2014-09-16 15:51:21,616 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-152:ctx-edb5bb0d) Seq 1-4070409638212796631: Executing request > 2014-09-16 15:51:21,645 DEBUG [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-76:ctx-abca3786) 9. The VM i-20-27-VM is in Stopping state > 2014-09-16 15:51:21,659 DEBUG [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-152:ctx-edb5bb0d) 9. The VM i-8-7-VM is in Stopping state > 2014-09-16 15:51:21,914 WARN [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-76:ctx-abca3786) callHostPlugin failed for cmd: > destroy_network_rules_for_vm with args vmName: i-20-27-VM, due to There was > a failure communicating with the plugin. > 2014-09-16 15:51:21,915 DEBUG [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-76:ctx-abca3786) Catch exception > com.cloud.utils.exception.CloudRuntimeException when stop VM:i-20-27-VM due > to com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for > cmd: destroy_network_rules_for_vm with args vmName: i-20-27-VM, due to There > was a failure communicating with the plugin. > 2014-09-16 15:51:21,917 DEBUG [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-76:ctx-abca3786) 10. The VM i-20-27-VM is in Stopped state > 2014-09-16 15:51:21,917 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-76:ctx-abca3786) Seq 1-4070409638212796630: Response Received: > 2014-09-16 15:51:21,917 DEBUG [c.c.a.t.Request] (DirectAgent-76:ctx-abca3786) > Seq 1-4070409638212796630: Processing: { Ans: , MgmtId: 125944753790399, > via: 1, Ver: v1, Flags: 10, > [{"com.cloud.agent.api.StopAnswer":{"platform":"viridian:true;acpi:1;apic:true;pae:true;nx:true","result":false,"details":"Catch > exception com.cloud.utils.exception.CloudRuntimeException when stop > VM:i-20-27-VM due to com.cloud.utils.exception.CloudRuntimeException: > callHostPlugin failed for cmd: destroy_network_rules_for_vm with args vmName: > i-20-27-VM, due to There was a failure communicating with the > plugin.","wait":0}}] } > 2014-09-16 15:51:21,917 DEBUG [c.c.a.t.Request] > (Work-Job-Executor-62:ctx-7afe827c job-207/job-209 ctx-9abbad0d) Seq > 1-4070409638212796630: Received: { Ans: , MgmtId: 125944753790399, via: 1, > Ver: v1, Flags: 10, { StopAnswer } } > 2014-09-16 15:51:21,923 WARN [c.c.v.VirtualMachineManagerImpl] > (Work-Job-Executor-62:ctx-7afe827c job-207/job-209 ctx-9abbad0d) Unable to > stop vm VM[User|i-20-27-VM] > 2014-09-16 15:51:21,926 INFO [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-152:ctx-edb5bb0d) Removed network rules for vm i-8-7-VM > 2014-09-16 15:51:21,929 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-62:ctx-7afe827c job-207/job-209 ctx-9abbad0d) VM state > transitted from :Stopping to Running with event: OperationFailedvm's original > host id: 1 new host id: 1 host id before state transition: 1 > 2014-09-16 15:51:21,930 ERROR [c.c.v.VmWorkJobHandlerProxy] > (Work-Job-Executor-62:ctx-7afe827c job-207/job-209 ctx-9abbad0d) Invocation > exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Unable > to stop VM[User|i-20-27-VM] > 2014-09-16 15:51:21,930 INFO [c.c.v.VmWorkJobHandlerProxy] > (Work-Job-Executor-62:ctx-7afe827c job-207/job-209 ctx-9abbad0d) Rethrow > exception com.cloud.utils.exception.CloudRuntimeException: Unable to stop > VM[User|i-20-27-VM] > 2014-09-16 15:51:21,930 DEBUG [c.c.v.VmWorkJobDispatcher] > (Work-Job-Executor-62:ctx-7afe827c job-207/job-209) Done with run of VM work > job: com.cloud.vm.VmWorkStop for VM 27, job origin: 207 > 2014-09-16 15:51:21,931 ERROR [c.c.v.VmWorkJobDispatcher] > (Work-Job-Executor-62:ctx-7afe827c job-207/job-209) Unable to complete > AsyncJobVO {id:209, userId: 2, accountId: 2, instanceType: null, instanceId: > null, cmd: com.cloud.vm.VmWorkStop, cmdInfo: > rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAAG3QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwA, > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: > null, initMsid: 125944753790399, completeMsid: null, lastUpdated: null, > lastPolled: null, created: Tue Sep 16 15:51:20 UTC 2014}, job origin:207 > com.cloud.utils.exception.CloudRuntimeException: Unable to stop > VM[User|i-20-27-VM] > at > com.cloud.vm.VirtualMachineManagerImpl.advanceStop(VirtualMachineManagerImpl.java:1523) > at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStop(VirtualMachineManagerImpl.java:1377) > at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStop(VirtualMachineManagerImpl.java:4594) > at sun.reflect.GeneratedMethodAccessor367.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) > at > com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4738) > at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:722) > 2014-09-16 15:51:21,959 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (Work-Job-Executor-62:ctx-7afe827c job-207/job-209) Complete async job-209, > jobStatus: FAILED, resultCode: 0, result: > rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFuZy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARMAAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9sYW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAPkpvYiBmYWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBVbmFibGUgdG8gc3RvcCBWTVtVc2VyfGktMjAtMjctVk1ddXIAHltMamF2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50OwJGKjw8_SI5AgAAeHAAAAAOc3IAG2phdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudGEJxZomNt2FAgAESQAKbGluZU51bWJlckwADmRlY2xhcmluZ0NsYXNzcQB-AARMAAhmaWxlTmFtZXEAfgAETAAKbWV0aG9kTmFtZXEAfgAEeHAAAABydAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JzcQB-AAsAAAH3dAA_b3JnLmFwYWNoZS5jbG91ZHN0YWNrLmZyYW1ld29yay5qb2JzLmltcGwuQXN5bmNKb2JNYW5hZ2VySW1wbCQ1dAAYQXN5bmNKb2JNYW5hZ2VySW1wbC5qYXZhdAAMcnVuSW5Db250ZXh0c3EAfgALAAAAMXQAPm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZSQxdAAbTWFuYWdlZENvbnRleHRSdW5uYWJsZS5qYXZhdAADcnVuc3EAfgALAAAAOHQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHNxAH4ACwAAAGd0AEBvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0cQB-ABp0AA9jYWxsV2l0aENvbnRleHRzcQB-AAsAAAA1cQB-AB1xAH4AGnQADnJ1bldpdGhDb250ZXh0c3EAfgALAAAALnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAWcQB-ABdzcQB-AAsAAAHMcQB-ABFxAH4AEnEAfgAXc3EAfgALAAAB13QALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAbc3EAfgALAAABTnQAJGphdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2skU3luY3QAD0Z1dHVyZVRhc2suamF2YXQACGlubmVyUnVuc3EAfgALAAAApnQAH2phdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2txAH4AKXEAfgAXc3EAfgALAAAEVnQAJ2phdmEudXRpbC5jb25jdXJyZW50LlRocmVhZFBvb2xFeGVjdXRvcnQAF1RocmVhZFBvb2xFeGVjdXRvci5qYXZhdAAJcnVuV29ya2Vyc3EAfgALAAACW3QALmphdmEudXRpbC5jb25jdXJyZW50LlRocmVhZFBvb2xFeGVjdXRvciRXb3JrZXJxAH4AL3EAfgAXc3EAfgALAAAC0nQAEGphdmEubGFuZy5UaHJlYWR0AAtUaHJlYWQuamF2YXEAfgAXc3IAJmphdmEudXRpbC5Db2xsZWN0aW9ucyRVbm1vZGlmaWFibGVMaXN0_A8lMbXsjhACAAFMAARsaXN0cQB-AAZ4cgAsamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZUNvbGxlY3Rpb24ZQgCAy173HgIAAUwAAWN0ABZMamF2YS91dGlsL0NvbGxlY3Rpb247eHBzcgATamF2YS51dGlsLkFycmF5TGlzdHiB0h2Zx2GdAwABSQAEc2l6ZXhwAAAAAHcEAAAAAHhxAH4AO3g > 2014-09-16 15:51:21,967 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (Work-Job-Executor-62:ctx-7afe827c job-207/job-209) Done executing > com.cloud.vm.VmWorkStop for job-209 > 2014-09-16 15:51:21,977 DEBUG [o.a.c.f.j.i.SyncQueueManagerImpl] > (Work-Job-Executor-62:ctx-7afe827c job-207/job-209) Sync queue (61) is > currently empty > 2014-09-16 15:51:21,978 INFO [o.a.c.f.j.i.AsyncJobMonitor] > (Work-Job-Executor-62:ctx-7afe827c job-207/job-209) Remove job-209 from job > monitoring > 2014-09-16 15:51:21,981 ERROR [c.c.a.ApiAsyncJobDispatcher] > (API-Job-Executor-74:ctx-cbfc3d27 job-207) Unexpected exception while > executing org.apache.cloudstack.api.command.admin.vm.StopVMCmdByAdmin > java.lang.RuntimeException: Job failed due to exception Unable to stop > VM[User|i-20-27-VM] > at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:722) > 2014-09-16 15:51:21,985 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (API-Job-Executor-74:ctx-cbfc3d27 job-207) Complete async job-207, jobStatus: > FAILED, resultCode: 530, result: > org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Job > failed due to exception Unable to stop VM[User|i-20-27-VM]"} > 2014-09-16 15:51:21,992 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (API-Job-Executor-74:ctx-cbfc3d27 job-207) Done executing > org.apache.cloudstack.api.command.admin.vm.StopVMCmdByAdmin for job-207 > 2014-09-16 15:51:21,995 INFO [o.a.c.f.j.i.AsyncJobMonitor] > (API-Job-Executor-74:ctx-cbfc3d27 job-207) Remove job-207 from job monitoring > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)