Sangeetha Hariharan created CLOUDSTACK-5395:
-----------------------------------------------
Summary: When
Key: CLOUDSTACK-5395
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5395
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Reporter: Sangeetha Hariharan
Steps to reproduce the problem:
Deploy 5 Vms in each of the hosts with 10 GB , so we start with 10 Vms.
We will be constantly writing to the ROOT volume.
Change the backup.snapshot.wait to 10 mts and restart management server.
Start concurrent snapshots for ROOT volumes of all the Vms.
After 10 mts , the snapshots fail. They are present in the database in
"CreatedOnPrimary" state.
Vhd entries from the primary store fails to be cleaned up once the
backsnapshpot job fails.
Expected Behavior:
We should be able to clean up the Vhd entries from the primary store when the
backsnapshpot job fails.
select * from snapshot_store_ref;
702 | 1 | 355 | 2013-12-06 01:25:43 | NULL | NULL |
Primary | 0 | 0 | 0 |
2eedb23e-6c3f-4cae-832b-8ddb67c1fc60 | Ready | 2
| 0 | 2013-12-06 01:25:44 | 81 |
| 703 | 1 | 356 | 2013-12-06 01:25:43 | NULL | NULL |
Primary | 0 | 0 | 0 |
9d88bc01-9406-41ad-a134-e74dc1457954 | Ready | 2
| 0 | 2013-12-06 01:26:12 | 80 |
| 704 | 1 | 357 | 2013-12-06 01:25:43 | NULL | NULL |
Primary | 0 | 0 | 0 |
2667f2bc-6086-4ec3-a88d-20811eabde91 | Ready | 2
| 0 | 2013-12-06 01:26:08 | 79 |
| 705 | 1 | 358 | 2013-12-06 01:25:44 | NULL | NULL |
Primary | 0 | 0 | 0 |
522b2296-6960-46f2-af7d-10ddfbede1da | Ready | 2
| 0 | 2013-12-06 01:26:45 | 78 |
| 706 | 1 | 359 | 2013-12-06 01:25:44 | NULL | NULL |
Primary | 0 | 0 | 0 |
3b94fa9d-a5a5-4441-8f9f-275dcef90368 | Ready | 2
| 0 | 2013-12-06 01:26:04 | 77 |
| 707 | 1 | 360 | 2013-12-06 01:25:44 | NULL | NULL |
Primary | 0 | 0 | 0 |
1ec1d5ef-177f-4da4-8464-f0c6d71a4e84 | Ready | 2
| 0 | 2013-12-06 01:25:59 | 76 |
| 708 | 1 | 361 | 2013-12-06 01:25:44 | NULL | NULL |
Primary | 0 | 0 | 0 |
324e7552-b42a-4660-90d6-62015a7a478e | Ready | 2
| 0 | 2013-12-06 01:26:21 | 75 |
| 709 | 1 | 362 | 2013-12-06 01:25:44 | NULL | NULL |
Primary | 0 | 0 | 0 |
65bd522c-c2c8-471a-be37-095558d058f2 | Ready | 2
| 0 | 2013-12-06 01:26:16 | 74 |
| 710 | 1 | 363 | 2013-12-06 01:25:44 | NULL | NULL |
Primary | 0 | 0 | 0 |
d45ca6c7-7284-4150-907c-9499e9737c47 | Ready | 2
| 0 | 2013-12-06 01:25:46 | 73 |
| 711 | 1 | 364 | 2013-12-06 01:25:44 | NULL | NULL |
Primary | 0 | 0 | 0 |
4422f362-0be5-4a10-b172-45678d56f807 | Ready | 2
| 0 | 2013-12-06 01:25:55 | 72 |
| 712 | 1 | 365 | 2013-12-06 01:25:44 | NULL | NULL |
Primary | 0 | 0 | 0 |
89ffd430-3c03-45d2-9c48-9384636b9cd8 | Ready | 2
| 0 | 2013-12-06 01:26:01 | 71 |
| 714 | 1 | 366 | 2013-12-06 01:25:45 | NULL | NULL |
Primary | 0 | 0 | 0 |
fca5545c-9b83-4bc1-abd2-dd1bc82b23bd | Ready | 2
| 0 | 2013-12-06 01:25:53 | 70 |
| 715 | 1 | 367 | 2013-12-06 01:25:45 | NULL | NULL |
Primary | 0 | 0 | 0 |
033d8f55-8895-40b8-a120-11b28fa1f96e | Ready | 2
| 0 | 2013-12-06 01:25:50 | 69 |
| 716 | 1 | 368 | 2013-12-06 01:25:45 | NULL | NULL |
Primary | 0 | 0 | 0 |
e4d02558-28c2-474e-a379-970b22f33f55 | Ready | 2
| 0 | 2013-12-06 01:26:23 | 68 |
| 717 | 1 | 369 | 2013-12-06 01:25:45 | NULL | NULL |
Primary | 0 | 0 | 0 |
6f7c1ca0-9877-44af-9f77-4db7b8efc934 | Ready | 2
| 0 | 2013-12-06 01:26:37 | 67 |
| 718 | 1 | 370 | 2013-12-06 01:25:45 | NULL | NULL |
Primary | 0 | 0 | 0 |
e3860504-7e37-416a-8216-c4a021f5e8bf | Ready | 2
| 0 | 2013-12-06 01:25:47 | 66 |
| 719 | 1 | 371 | 2013-12-06 01:25:45 | NULL | NULL |
Primary | 0 | 0 | 0 |
48c2a9bd-e7ce-429f-be73-ef9c19964ad3 | Ready | 2
| 0 | 2013-12-06 01:26:17 | 65 |
| 720 | 1 | 372 | 2013-12-06 01:25:45 | NULL | NULL |
Primary | 0 | 0 | 0 |
3eabb081-d0d9-41d6-9c11-00e002031461 | Ready | 2
| 0 | 2013-12-06 01:25:51 | 64 |
| 721 | 1 | 373 | 2013-12-06 01:25:46 | NULL | NULL |
Primary | 0 | 0 | 0 |
c9968fbb-0876-4846-94aa-e44ea7e3143d | Ready | 2
| 0 | 2013-12-06 01:26:06 | 63 |
| 723 | 1 | 374 | 2013-12-06 01:25:46 | NULL | NULL |
Primary | 0 | 0 | 0 |
72fdec4a-c2f1-48e1-80bf-806b6db46820 | Ready | 2
| 0 | 2013-12-06 01:26:02 | 62 |
| 742 | 1 | 375 | 2013-12-06 01:40:43 | NULL | NULL |
Primary | 0 | 0 | 0 |
8a692b5a-569e-473c-83c4-dc32c09c8491 | Ready | 2
| 0 | 2013-12-06 01:40:47 | 60 |
+-----+----------+-------------+---------------------+--------------+--------+------------+------+---------------+--------------------+-----------------------------------------------------+-----------+--------------+---------+---------------------+-----------+
2013-12-05 20:51:41,212 DEBUG [c.c.s.s.SnapshotManagerImpl]
(Job-Executor-42:ctx-c20be5fc ctx-5f8cfbf3) Failed to create snapshot
com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot Failed due to
Async 600 seconds timeout for task com.xensource.xenapi.Task@8b2ca24f
at
org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:275)
at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:135)
at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:294)
at
com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:951)
at sun.reflect.GeneratedMethodAccessor405.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy161.takeSnapshot(Unknown Source)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1341)
at
com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:1408)
at sun.reflect.GeneratedMethodAccessor404.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy233.takeSnapshot(Unknown Source)
at
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:181)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
at
com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
at
com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:520)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
2013-12-05 20:51:41,217 DEBUG [o.a.c.s.v.VolumeServiceImpl]
(Job-Executor-42:ctx-c20be5fc ctx-5f8cfbf3) Take snapshot: 60 failed
com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot
at
com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:972)
at sun.reflect.GeneratedMethodAccessor405.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy161.takeSnapshot(Unknown Source)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1341)
at
com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:1408)
at sun.reflect.GeneratedMethodAccessor404.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy233.takeSnapshot(Unknown Source)
at
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:181)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
at
com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
at
com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:520)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot
Failed due to Async 600 seconds timeout for task
com.xensource.xenapi.Task@8b2ca24f
at
org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:275)
at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:135)
at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:294)
at
com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:951)
--
This message was sent by Atlassian JIRA
(v6.1#6144)
