Sangeetha Hariharan created CLOUDSTACK-5736:
-----------------------------------------------
Summary: KVM - Parallel Snapshots - Snapshots reported as being in
"BackedUp" state even though a Nullpointer exception is seen during post
processing of snapshot.
Key: CLOUDSTACK-5736
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5736
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Management Server
Affects Versions: 4.3.0
Environment: Build from 4.3
Reporter: Sangeetha Hariharan
Priority: Critical
Fix For: 4.3.0
Set up:
Advanced zone set up with 2 KVM (rhel 63) hosts.
Steps to reproduce the problem:
Deploy about 17 Vms in both hosts.
Start "Hourly" snapshot for all the ROOT volumes of these Vms.
Snapshots were reported as being in "BackedUp" state even though Nullpointer
exception is seen during post processing of snapshot.
Following exception seen in management server logs:
014-01-02 19:57:53,560 DEBUG [c.c.s.s.SnapshotManagerImpl]
(Job-Executor-166:ctx-c0f157b4 ctx-2654c4f7) post process snapshot failed
java.lang.NullPointerException
at
com.cloud.storage.snapshot.SnapshotManagerImpl.postCreateSnapshot(SnapshotManagerImpl.java:374)
at
com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:954)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy160.takeSnapshot(Unknown Source)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1371)
at
com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:1723)
at
com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:2478)
at sun.reflect.GeneratedMethodAccessor524.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy195.handleVmWorkJob(Unknown Source)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:522)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
mysql> select id,volume_id,name,status from snapshots where status="BackedUp";
+----+-----------+-------------------------------------------------+----------+
| id | volume_id | name | status |
+----+-----------+-------------------------------------------------+----------+
| 4 | 22 | TestVM-tiny-host-0ps-0-6_ROOT-22_20140102234418 | BackedUp |
| 5 | 21 | TestVM-tiny-host-0ps-0-5_ROOT-21_20140102234418 | BackedUp |
| 6 | 20 | TestVM-tiny-host-0ps-0-4_ROOT-20_20140102234418 | BackedUp |
| 7 | 19 | TestVM-tiny-host-0ps-0-3_ROOT-19_20140102234418 | BackedUp |
| 8 | 18 | TestVM-tiny-host-0ps-0-2_ROOT-18_20140102234418 | BackedUp |
| 9 | 17 | TestVM-tiny-host-0ps-0-1_ROOT-17_20140102234418 | BackedUp |
| 10 | 16 | TestVM-tiny-host-0ps-0-0_ROOT-16_20140102234419 | BackedUp |
| 11 | 14 | TestVM-1_ROOT-14_20140102234419 | BackedUp |
| 12 | 31 | TestVM-tiny-host-1ps-0-5_ROOT-31_20140102234419 | BackedUp |
| 16 | 27 | TestVM-tiny-host-1ps-0-1_ROOT-27_20140102234419 | BackedUp |
| 17 | 26 | TestVM-tiny-host-1ps-0-0_ROOT-26_20140102234419 | BackedUp |
| 18 | 25 | TestVM-tiny-host-0ps-0-9_ROOT-25_20140102234419 | BackedUp |
| 19 | 24 | TestVM-tiny-host-0ps-0-8_ROOT-24_20140102234419 | BackedUp |
+----+-----------+-------------------------------------------------+----------+
14 rows in set (0.00 sec)
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
