Sangeetha Hariharan created CLOUDSTACK-5430:
-----------------------------------------------
Summary: KVM - Primary store down - Not abel to start Vms/take
snapshots after the primary store is brought down and brough back up again.
Key: CLOUDSTACK-5430
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5430
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Management Server
Affects Versions: 4.3.0
Environment: Build from 4.3
Reporter: Sangeetha Hariharan
Assignee: edison su
Priority: Critical
Fix For: 4.3.0
KVM - Primary store down - Not abel to start Vms/take snapshots after the
primary store is brought down and brough back up again.
Set up:
Advanced zone with KVM (RHEL 6.3) hosts.
Steps to reproduce the problem:
1. Deploy few Vms in each of the hosts with 10 GB ROOT volume size , so we
start with 10 Vms.
2. Create snaposhot for ROOT volumes.
3. When snapshot is still in progress , Make the primary storage unavailable
for 10 mts.
This results in the KVM hosts to reboot.
But reboot of KVM host is not successful.It is stuck at trying to unmount nfs
mount points. This is tracked in CLOUDSTACK-5429.
Stop and start KM hosts manually to workaround this problem.
At this point all the Vms are marked as "Stopped" state in CloudStack.
4. Now make the primary store available.
5. Attempt to start the VM.
It fails to start with the following exception:
2013-12-09 20:35:55,891 DEBUG [c.c.a.t.Request] (AgentManager-Handler-2:null)
Seq 2-1983250480: Processing: { Ans
: , MgmtId: 82324189320212, via: 2, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":false,"details":"
java.lang.NullPointerException\n\tat
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtCom
putingResource.java:2488)\n\tat
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtC
omputingResource.java:1260)\n\tat
com.cloud.agent.Agent.processRequest(Agent.java:498)\n\tat com.cloud.agent.Agent
$AgentRequestHandler.doTask(Agent.java:806)\n\tat
com.cloud.utils.nio.Task.run(Task.java:83)\n\tat java.util.concu
rrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)\n\tat
java.util.concurrent.ThreadPoolExecutor$Wor
ker.run(ThreadPoolExecutor.java:603)\n\tat
java.lang.Thread.run(Thread.java:679)\n","wait":0}}] }
2013-12-09 20:35:55,891 DEBUG [c.c.a.t.Request] (StatsCollector-3:ctx-f0d35c47)
Seq 2-1983250480: Received: { Ans
: , MgmtId: 82324189320212, via: 2, Ver: v1, Flags: 10, { Answer } }
2013-12-09 20:35:56,939 DEBUG [c.c.a.ApiServlet]
(catalina-exec-13:ctx-35adede4) ===START=== 10.216.50.147 -- GET
command=queryAsyncJobResult&jobId=489806e9-96f9-4940-9ea0-6bd9516aabb0&response=json&sessionkey=qRSeXYRCfc1PSAXc
omRT8ue1f%2BE%3D&_=1386639381768
2013-12-09 20:35:56,953 DEBUG [c.c.a.ApiServlet] (catalina-exec-13:ctx-35adede4
ctx-065180b8) ===END=== 10.216.50
.147 -- GET
command=queryAsyncJobResult&jobId=489806e9-96f9-4940-9ea0-6bd9516aabb0&response=json&sessionkey=qRSeX
YRCfc1PSAXcomRT8ue1f%2BE%3D&_=1386639381768
2013-12-09 20:35:59,322 DEBUG [c.c.a.t.Request] (AgentManager-Handler-14:null)
Seq 1-539557989: Processing: { Ans
: , MgmtId: 82324189320212, via: 1, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":false,"details":"
java.lang.NullPointerException\n\tat
com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.disconnectPhysicalDisk
sViaVmSpec(KVMStoragePoolManager.java:181)\n\tat
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execut
e(LibvirtComputingResource.java:3672)\n\tat
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1282)\n\tat
com.cloud.agent.Agent.processRequest(Agent.java:498)\n\tat
com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:806)\n\tat
com.cloud.utils.nio.Task.run(Task.java:83)\n\tat
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)\n\tat
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)\n\tat
java.lang.Thread.run(Thread.java:679)\n","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped
by previous
failure","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped
by previous
failure","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped
by previous
failure","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped
by previous
failure","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped
by previous failure","wait":0}}] }
2013-12-09 20:35:59,322 DEBUG [c.c.a.t.Request] (Job-Executor-26:ctx-0382e21d
ctx-d8f9d323) Seq 1-539557989: Received: { Ans: , MgmtId: 82324189320212, via:
1, Ver: v1, Flags: 10, { Answer, Answer, Answer, Answer, Answer, Answer } }
6. Attempting to take snapshots also fails with following exception:
2013-12-09 20:54:10,509 DEBUG [c.c.a.t.Request] (AgentManager-Handler-10:null)
Seq 2-1983250525: Processing: { An
s: , MgmtId: 82324189320212, via: 2, Ver: v1, Flags: 10,
[{"org.apache.cloudstack.storage.command.CreateObjectAnsw
er":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException:
java.lang.NullPointerException","w
ait":0}}] }
2013-12-09 20:54:10,509 DEBUG [c.c.a.t.Request] (Job-Executor-34:ctx-eb237191
ctx-20bb478f) Seq 2-1983250525: Rece
ived: { Ans: , MgmtId: 82324189320212, via: 2, Ver: v1, Flags: 10, {
CreateObjectAnswer } }
2013-12-09 20:54:10,509 DEBUG [o.a.c.s.s.SnapshotServiceImpl]
(Job-Executor-34:ctx-eb237191 ctx-20bb478f) create s
napshot TestVM-tiny-host-0ps-0-4_ROOT-49_20131210014410 failed:
com.cloud.utils.exception.CloudRuntimeException: j
ava.lang.NullPointerException
2013-12-09 20:54:10,519 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
(Job-Executor-34:ctx-eb237191 ctx-20bb478f) Fa
iled to take snapshot: com.cloud.utils.exception.CloudRuntimeException:
java.lang.NullPointerException
2013-12-09 20:54:10,536 DEBUG [c.c.s.s.SnapshotManagerImpl]
(Job-Executor-34:ctx-eb237191 ctx-20bb478f) Failed to
create snapshot
com.cloud.utils.exception.CloudRuntimeException:
com.cloud.utils.exception.CloudRuntimeException: java.lang.NullPo
interException
at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy
.java:281)
at
com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:951)
at sun.reflect.GeneratedMethodAccessor230.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation
.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:15
0)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java
:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:17
2)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy161.takeSnapshot(Unknown Source)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1341)
at
com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:1461)
at sun.reflect.GeneratedMethodAccessor229.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy233.takeSnapshot(Unknown Source)
at
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:181)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
at
com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
at
com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:520)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
2013-12-09 20:54:10,544 DEBUG [o.a.c.s.v.VolumeServiceImpl]
(Job-Executor-34:ctx-eb237191 ctx-20bb478f) Take snapshot: 49 failed
com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
