Chandan Purushothama created CLOUDSTACK-6459:
------------------------------------------------
Summary: Unable to enable maintenance mode on a Primary storage
that crashed
Key: CLOUDSTACK-6459
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-6459
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Management Server
Affects Versions: 4.4.0
Reporter: Chandan Purushothama
Priority: Critical
Fix For: 4.4.0
Primary storage in my setup got powered off. I am not able to enable
maintenance mode on this primary storage.
Enabling maintenance mode on the primary storage fails with the following
error. It eventually timed out after trying many times
2014-04-18 16:43:50,020 DEBUG [c.c.a.ApiServlet] (catalina-exec-1:ctx-bd92e323
ctx-7d4c2498) ===END=== 10.214.5.40 -- GET
command=queryAsyncJobResult&jobId=62f6830a-c409-4449-a9c5-6a35b7b9fbed&response=json&sessionkey=WBpwG%2FryPRNNB1GRuHqam1zbtS8%3D&_=1397865006850
2014-04-18 16:43:50,495 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-9:null) SeqA 2-792: Processing Seq 2-792: { Cmd ,
MgmtId: -1, via: 2, Ver: v1, Flags: 11,
[{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":1,"_loadInfo":"{\n
\"connections\": []\n}","wait":0}}] }
2014-04-18 16:43:50,504 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-9:null) SeqA 2-792: Sending Seq 2-792: { Ans: , MgmtId:
6638073284439, via: 2, Ver: v1, Flags: 100010,
[{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
2014-04-18 16:43:52,539 WARN [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-143:ctx-16ea61bc) Async 600 seconds timeout for task
com.xensource.xenapi.Task@8aa497e8
2014-04-18 16:43:52,563 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-143:ctx-16ea61bc) unable to destroy
task(com.xensource.xenapi.Task@8aa497e8) on
host(0d2ea73b-12c0-433c-b1c3-e1f193e68f6e) due to You gave an invalid object
reference. The object may have recently been deleted. The class parameter
gives the type of reference given, and the handle parameter echoes the bad
value given.
2014-04-18 16:43:52,564 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-143:ctx-16ea61bc) Catch exception
com.cloud.utils.exception.CloudRuntimeException when stop VM:i-3-3-DR due to
com.cloud.utils.exception.CloudRuntimeException: Shutdown VM catch
HandleInvalid and VM is not in HALTED state
2014-04-18 16:43:52,569 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-143:ctx-16ea61bc) 10. The VM i-3-3-DR is in Running state
2014-04-18 16:43:52,572 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-143:ctx-16ea61bc) Seq 1-2385781902599520418: Response Received:
2014-04-18 16:43:52,573 DEBUG [c.c.a.t.Request] (DirectAgent-143:ctx-16ea61bc)
Seq 1-2385781902599520418: Processing: { Ans: , MgmtId: 6638073284439, via: 1,
Ver: v1, Flags: 10,
[{"com.cloud.agent.api.StopAnswer":{"platform":"viridian:true;acpi:1;apic:true;pae:true;nx:true","result":false,"details":"Catch
exception com.cloud.utils.exception.CloudRuntimeException when stop
VM:i-3-3-DR due to com.cloud.utils.exception.CloudRuntimeException: Shutdown VM
catch HandleInvalid and VM is not in HALTED state","wait":0}}] }
2014-04-18 16:43:52,576 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-2:job-30/job-31 ctx-191e1825) Seq 1-2385781902599520418:
Received: { Ans: , MgmtId: 6638073284439, via: 1, Ver: v1, Flags: 10, {
StopAnswer } }
2014-04-18 16:43:52,591 WARN [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-2:job-30/job-31 ctx-191e1825) Unable to stop vm
VM[User|i-3-3-DR]
2014-04-18 16:43:52,616 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-2:job-30/job-31 ctx-191e1825) VM state transitted from
:Stopping to Running with event: OperationFailedvm's original host id: 1 new
host id: 1 host id before state transition: 1
2014-04-18 16:43:52,616 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-2:job-30/job-31 ctx-191e1825) Invocation exception, caused
by: com.cloud.utils.exception.CloudRuntimeException: Unable to stop
VM[User|i-3-3-DR]
2014-04-18 16:43:52,617 INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-2:job-30/job-31 ctx-191e1825) Rethrow exception
com.cloud.utils.exception.CloudRuntimeException: Unable to stop
VM[User|i-3-3-DR]
2014-04-18 16:43:52,617 DEBUG [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-2:job-30/job-31) Done with run of VM work job:
com.cloud.vm.VmWorkStop for VM 3, job origin: 30
2014-04-18 16:43:52,617 ERROR [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-2:job-30/job-31) Unable to complete AsyncJobVO {id:31,
userId: 2, accountId: 2, instanceType: null, instanceId: null, cmd:
com.cloud.vm.VmWorkStop, cmdInfo:
rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAAA3QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwA,
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
null, initMsid: 6638073284439, completeMsid: null, lastUpdated: null,
lastPolled: null, created: Fri Apr 18 16:33:50 PDT 2014}, job origin:30
com.cloud.utils.exception.CloudRuntimeException: Unable to stop
VM[User|i-3-3-DR]
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStop(VirtualMachineManagerImpl.java:1533)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStop(VirtualMachineManagerImpl.java:1385)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStop(VirtualMachineManagerImpl.java:5175)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5309)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:495)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:452)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-04-18 16:43:52,642 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-2:job-30/job-31) Complete async job-31, jobStatus: FAILED,
resultCode: 0, result:
rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFuZy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARMAAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9sYW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAPEpvYiBmYWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBVbmFibGUgdG8gc3RvcCBWTVtVc2VyfGktMy0zLURSXXVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5kZWNsYXJpbmdDbGFzc3EAfgAETAAIZmlsZU5hbWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAAcnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9ic3EAfgALAAAB73QAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdvcmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQADHJ1bkluQ29udGV4dHNxAH4ACwAAADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnNxAH4ACwAAADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhbGxzcQB-AAsAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250ZXh0c3EAfgALAAAANXEAfgAdcQB-ABp0AA5ydW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4AFnEAfgAXc3EAfgALAAABxHEAfgARcQB-ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AG3NxAH4ACwAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZhcQB-ABdzcQB-AAsAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdzcQB-AAsAAALodAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhyACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QAFkxqYXZhL3V0aWwvQ29sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleHAAAAAAdwQAAAAAeHEAfgA4eA
2014-04-18 16:43:52,659 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-2:job-30/job-31) Done executing com.cloud.vm.VmWorkStop for
job-31
2014-04-18 16:43:52,680 ERROR [c.c.s.StoragePoolAutomationImpl]
(API-Job-Executor-3:job-30 ctx-51f6a0d8) Exception in enabling primary storage
maintenance:
java.lang.RuntimeException: Job failed due to exception Unable to stop
VM[User|i-3-3-DR]
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:495)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:452)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-04-18 16:43:52,687 DEBUG [o.a.c.f.j.i.SyncQueueManagerImpl]
(Work-Job-Executor-2:job-30/job-31) Sync queue (3) is currently empty
2014-04-18 16:43:52,688 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(Work-Job-Executor-2:job-30/job-31) Remove job-31 from job monitoring
2014-04-18 16:43:52,693 ERROR [c.c.a.ApiAsyncJobDispatcher]
(API-Job-Executor-3:job-30) Unexpected exception while executing
org.apache.cloudstack.api.command.admin.storage.PreparePrimaryStorageForMaintenanceCmd
com.cloud.utils.exception.CloudRuntimeException: Job failed due to exception
Unable to stop VM[User|i-3-3-DR]
at
com.cloud.storage.StoragePoolAutomationImpl.maintain(StoragePoolAutomationImpl.java:276)
at
org.apache.cloudstack.storage.datastore.lifecycle.CloudStackPrimaryDataStoreLifeCycleImpl.maintain(CloudStackPrimaryDataStoreLifeCycleImpl.java:459)
at
com.cloud.storage.StorageManagerImpl.preparePrimaryStorageForMaintenance(StorageManagerImpl.java:1203)
at
com.cloud.storage.StorageManagerImpl.preparePrimaryStorageForMaintenance(StorageManagerImpl.java:176)
at
org.apache.cloudstack.api.command.admin.storage.PreparePrimaryStorageForMaintenanceCmd.execute(PreparePrimaryStorageForMaintenanceCmd.java:103)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:134)
at
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:100)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:495)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:452)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-04-18 16:43:52,695 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-3:job-30) Complete async job-30, jobStatus: FAILED,
resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Job
failed due to exception Unable to stop VM[User|i-3-3-DR]"}
2014-04-18 16:43:52,703 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-3:job-30) Done executing
org.apache.cloudstack.api.command.admin.storage.PreparePrimaryStorageForMaintenanceCmd
for job-30
2014-04-18 16:43:52,709 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-3:job-30) Remove job-30 from job monitoring
2014-04-18 16:43:52,952 DEBUG [c.c.a.ApiServlet] (catalina-exec-7:ctx-ce58b540)
===START=== 10.214.5.40 -- GET
command=queryAsyncJobResult&jobId=62f6830a-c409-4449-a9c5-6a35b7b9fbed&response=json&sessionkey=WBpwG%2FryPRNNB1GRuHqam1zbtS8%3D&_=1397865009850
2014-04-18 16:43:52,994 DEBUG [c.c.a.ApiServlet] (catalina-exec-7:ctx-ce58b540
ctx-747fe328) ===END=== 10.214.5.40 -- GET
command=queryAsyncJobResult&jobId=62f6830a-c409-4449-a9c5-6a35b7b9fbed&response=json&sessionkey=WBpwG%2FryPRNNB1GRuHqam1zbtS8%3D&_=1397865009850
--
This message was sent by Atlassian JIRA
(v6.2#6252)
