Sangeetha Hariharan created CLOUDSTACK-5262:
-----------------------------------------------
Summary: Few of the snapshot creation from ROOT volume fails when
there are concurrent snapshots in progress.
Key: CLOUDSTACK-5262
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5262
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Management Server
Affects Versions: 4.3.0
Environment: Build from 4.3
Reporter: Sangeetha Hariharan
Priority: Critical
Fix For: 4.3.0
Steps to reproduce the problem:
Set up - Advanced zone with 2 Xenserver 6.2 hosts
Deploy 10 Vms in each of the hosts , so we start with 20 Vms.
We are constantly writing to the ROOT volume ( print timestamp every 1 minute).
Start concurrent snapshots for ROOT volumes for all the Vms.
Few of the snapshot jobs ( 4 of them) are failing due to the following
exception:
2013-11-25 08:16:14,239 DEBUG [c.c.a.t.Request] (Job-Executor-165:ctx-bae63a1a
ctx-4f637cc6) Seq 2-756819795: Sending { Cmd , MgmtId: 112516401760401, via:
2(Rack3Host23.lab.vmops.com), Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"ee56cec0-632d-469f-9326-d683e9fc1425","volume":{"uuid":"aee2805e-6394-47a6-9360-65829ce61929","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"babe1c7d-3154-3b46-addc-837ca2b7f07d","id":1,"poolType":"NetworkFilesystem","host":"10.223.110.231","path":"/export/home/sangeetha/felton/xen/primary","port":2049,"url":"NetworkFilesystem://10.223.110.231//export/home/sangeetha/felton/xen/primary/?ROLE=Primary&STOREUUID=babe1c7d-3154-3b46-addc-837ca2b7f07d"}},"name":"ROOT-73","size":21474836480,"path":"d1988b67-9f8f-4685-9bfd-a171238e8abd","volumeId":73,"vmName":"i-9-73-MyTestVM","accountId":9,"format":"VHD","id":73,"deviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"d3975b00-d9c3-418a-9a39-f09139754a32","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"babe1c7d-3154-3b46-addc-837ca2b7f07d","id":1,"poolType":"NetworkFilesystem","host":"10.223.110.231","path":"/export/home/sangeetha/felton/xen/primary","port":2049,"url":"NetworkFilesystem://10.223.110.231//export/home/sangeetha/felton/xen/primary/?ROLE=Primary&STOREUUID=babe1c7d-3154-3b46-addc-837ca2b7f07d"}},"vmName":"i-9-73-MyTestVM","name":"TestVM-tiny-host-0ps-0-3_ROOT-73_20131125131535","hypervisorType":"XenServer","id":296,"quiescevm":false}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/9/73","volume":{"uuid":"aee2805e-6394-47a6-9360-65829ce61929","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"babe1c7d-3154-3b46-addc-837ca2b7f07d","id":1,"poolType":"NetworkFilesystem","host":"10.223.110.231","path":"/export/home/sangeetha/felton/xen/primary","port":2049,"url":"NetworkFilesystem://10.223.110.231//export/home/sangeetha/felton/xen/primary/?ROLE=Primary&STOREUUID=babe1c7d-3154-3b46-addc-837ca2b7f07d"}},"name":"ROOT-73","size":21474836480,"path":"d1988b67-9f8f-4685-9bfd-a171238e8abd","volumeId":73,"vmName":"i-9-73-MyTestVM","accountId":9,"format":"VHD","id":73,"deviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"snapshots/9/73/20ebe402-5c2f-4a61-a0c4-525d97ebfa71","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.223.110.233:/export/home/sangeetha/felton/xen/secondary","_role":"Image"}},"vmName":"i-9-73-MyTestVM","name":"TestVM-tiny-host-0ps-0-3_ROOT-73_20131125131535","hypervisorType":"XenServer","id":296,"quiescevm":false}},"executeInSequence":false,"wait":21600}}]
}
2013-11-25 08:16:14,239 DEBUG [c.c.a.t.Request] (Job-Executor-165:ctx-bae63a1a
ctx-4f637cc6) Seq 2-756819795: Executing: { Cmd , MgmtId: 112516401760401,
via: 2(Rack3Host23.lab.vmops.com), Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"ee56cec0-632d-469f-9326-d683e9fc1425","volume":{"uuid":"aee2805e-6394-47a6-9360-65829ce61929","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"babe1c7d-3154-3b46-addc-837ca2b7f07d","id":1,"poolType":"NetworkFilesystem","host":"10.223.110.231","path":"/export/home/sangeetha/felton/xen/primary","port":2049,"url":"NetworkFilesystem://10.223.110.231//export/home/sangeetha/felton/xen/primary/?ROLE=Primary&STOREUUID=babe1c7d-3154-3b46-addc-837ca2b7f07d"}},"name":"ROOT-73","size":21474836480,"path":"d1988b67-9f8f-4685-9bfd-a171238e8abd","volumeId":73,"vmName":"i-9-73-MyTestVM","accountId":9,"format":"VHD","id":73,"deviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"d3975b00-d9c3-418a-9a39-f09139754a32","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"babe1c7d-3154-3b46-addc-837ca2b7f07d","id":1,"poolType":"NetworkFilesystem","host":"10.223.110.231","path":"/export/home/sangeetha/felton/xen/primary","port":2049,"url":"NetworkFilesystem://10.223.110.231//export/home/sangeetha/felton/xen/primary/?ROLE=Primary&STOREUUID=babe1c7d-3154-3b46-addc-837ca2b7f07d"}},"vmName":"i-9-73-MyTestVM","name":"TestVM-tiny-host-0ps-0-3_ROOT-73_20131125131535","hypervisorType":"XenServer","id":296,"quiescevm":false}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/9/73","volume":{"uuid":"aee2805e-6394-47a6-9360-65829ce61929","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"babe1c7d-3154-3b46-addc-837ca2b7f07d","id":1,"poolType":"NetworkFilesystem","host":"10.223.110.231","path":"/export/home/sangeetha/felton/xen/primary","port":2049,"url":"NetworkFilesystem://10.223.110.231//export/home/sangeetha/felton/xen/primary/?ROLE=Primary&STOREUUID=babe1c7d-3154-3b46-addc-837ca2b7f07d"}},"name":"ROOT-73","size":21474836480,"path":"d1988b67-9f8f-4685-9bfd-a171238e8abd","volumeId":73,"vmName":"i-9-73-MyTestVM","accountId":9,"format":"VHD","id":73,"deviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"snapshots/9/73/20ebe402-5c2f-4a61-a0c4-525d97ebfa71","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.223.110.233:/export/home/sangeetha/felton/xen/secondary","_role":"Image"}},"vmName":"i-9-73-MyTestVM","name":"TestVM-tiny-host-0ps-0-3_ROOT-73_20131125131535","hypervisorType":"XenServer","id":296,"quiescevm":false}},"executeInSequence":false,"wait":21600}}]
}
2013-11-25 08:16:14,239 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-364:ctx-a502b0ee) Seq 2-756819795: Executing request
2013-11-25 08:16:16,738 WARN [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-364:ctx-a502b0ee) callHostPlugin failed for cmd: backupSnapshot
with args secondaryStorageMountPath:
10.223.110.233:/export/home/sangeetha/felton/xen/secondary, localMountPoint:
/var/run/cloud_mount/e8eabfa9-3012-36f4-adf5-895bbf085947, backupUuid:
07f55f61-e1a5-4880-a9a0-37e063e88e39, path: snapshots/9/73, prevBackupUuid:
snapshots/9/73/20ebe402-5c2f-4a61-a0c4-525d97ebfa71, snapshotUuid:
ee56cec0-632d-469f-9326-d683e9fc1425, isISCSI: false, primaryStorageSRUuid:
69ec4a37-f19c-8951-8b52-ebe3437f4b5e, due to HandleInvalid clazz:task,
handle:OpaqueRef:2d1621a0-7c9e-ea80-48ef-c7114ce3a63d
2013-11-25 08:16:16,740 WARN [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-364:ctx-a502b0ee) unable to destroy
task(com.xensource.xenapi.Task@4b81265b) on
host(b8c15cde-f075-487b-86bc-9ef564900887) due to
You gave an invalid object reference. The object may have recently been
deleted. The class parameter gives the type of reference given, and the handle
parameter echoes the bad value given.
at com.xensource.xenapi.Types.checkResponse(Types.java:209)
at com.xensource.xenapi.Connection.dispatch(Connection.java:368)
at
com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:909)
at com.xensource.xenapi.Task.destroy(Task.java:616)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPluginAsync(CitrixResourceBase.java:4067)
at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.backupSnapshot(XenServerStorageProcessor.java:1131)
at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.backupSnapshot(XenServerStorageProcessor.java:1320)
at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:90)
at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:50)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:612)
at
com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:59)
at
com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:106)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
2013-11-25 08:16:17,097 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-2:null) SeqA 3-37675: Processing Seq 3-37675: { Cmd ,
MgmtId: -1, via: 3, Ver: v1, Flags: 11,
[{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":1,"_loadInfo":"{\n
\"connections\": []\n}","wait":0}}] }
2013-11-25 08:16:17,100 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-2:null) SeqA 3-37675: Sending Seq 3-37675: { Ans: ,
MgmtId: 112516401760401, via: 3, Ver: v1, Flags: 100010,
[{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
2013-11-25 08:16:17,755 WARN [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-364:ctx-a502b0ee) Could not copy backupUuid: null from primary
storage 69ec4a37-f19c-8951-8b52-ebe3437f4b5e to secondary storage
10.223.110.233:/export/home/sangeetha/felton/xen/secondary due to null
2013-11-25 08:16:17,755 WARN [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-364:ctx-a502b0ee) BackupSnapshot Failed due to Could not copy
backupUuid: null from primary storage 69ec4a37-f19c-8951-8b52-ebe3437f4b5e to
secondary storage 10.223.110.233:/export/home/sangeetha/felton/xen/secondary
due to null
com.cloud.utils.exception.CloudRuntimeException: Could not copy backupUuid:
null from primary storage 69ec4a37-f19c-8951-8b52-ebe3437f4b5e to secondary
storage 10.223.110.233:/export/home/sangeetha/felton/xen/secondary due to null
at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.backupSnapshot(XenServerStorageProcessor.java:1160)
at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.backupSnapshot(XenServerStorageProcessor.java:1320)
at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:90)
at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:50)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:612)
at
com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:59)
at
com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:106)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
2013-11-25 08:16:17,756 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-364:ctx-a502b0ee) Seq 2-756819795: Response Received:
2013-11-25 08:16:17,756 DEBUG [c.c.a.t.Request] (DirectAgent-364:ctx-a502b0ee)
Seq 2-756819795: Processing: { Ans: , MgmtId: 112516401760401, via: 2, Ver:
v1, Flags: 10,
[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"BackupSnapshot
Failed due to Could not copy backupUuid: null from primary storage
69ec4a37-f19c-8951-8b52-ebe3437f4b5e to secondary storage
10.223.110.233:/export/home/sangeetha/felton/xen/secondary due to
null","wait":0}}] }
2013-11-25 08:16:17,756 DEBUG [c.c.a.t.Request] (Job-Executor-165:ctx-bae63a1a
ctx-4f637cc6) Seq 2-756819795: Received: { Ans: , MgmtId: 112516401760401,
via: 2, Ver: v1, Flags: 10, { CopyCmdAnswer } }
2013-11-25 08:16:17,768 DEBUG [c.c.s.s.SnapshotManagerImpl]
(Job-Executor-165:ctx-bae63a1a ctx-4f637cc6) Failed to create snapshot
com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot Failed due to
Could not copy backupUuid: null from primary storage
69ec4a37-f19c-8951-8b52-ebe3437f4b5e to secondary storage
10.223.110.233:/export/home/sangeetha/felton/xen/secondary due to null
at
org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:275)
at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:138)
at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:289)
at
com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:951)
at sun.reflect.GeneratedMethodAccessor444.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy160.takeSnapshot(Unknown Source)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1342)
at
com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:1402)
at sun.reflect.GeneratedMethodAccessor443.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy232.takeSnapshot(Unknown Source)
at
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:181)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
at
com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
at
com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:520)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
2013-11-25 08:16:17,775 DEBUG [o.a.c.s.v.VolumeServiceImpl]
(Job-Executor-165:ctx-bae63a1a ctx-4f637cc6) Take snapshot: 73 failed
com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot
at
com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:972)
at sun.reflect.GeneratedMethodAccessor444.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy160.takeSnapshot(Unknown Source)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1342)
at
com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:1402)
at sun.reflect.GeneratedMethodAccessor443.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy232.takeSnapshot(Unknown Source)
at
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:181)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
at
com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
at
com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:520)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot
Failed due to Could not copy backupUuid: null from primary storage
69ec4a37-f19c-8951-8b52-ebe3437f4b5e to secondary storage
10.223.110.233:/export/home/sangeetha/felton/xen/secondary due to null
at
org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:275)
at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:138)
at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:289)
at
com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:951)
... 42 more
2013-11-25 08:16:17,776 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Job-Executor-165:ctx-bae63a1a) Complete async job-631, jobStatus: FAILED,
resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
to create snapshot due to an internal error creating snapshot for volume 73"}
mysql> select job_cmd , account_id , job_result, job_result_code, job_status
from async_job where job_status=2 and
job_cmd="org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd";
+-------------------------------------------------------------------+------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------+
| job_cmd |
account_id | job_result
| job_result_code | job_status |
+-------------------------------------------------------------------+------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------+
| org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd |
9 |
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
to create snapshot due to an internal error creating snapshot for volume 73"}
| 530 | 2 |
| org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd |
9 |
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
to create snapshot due to an internal error creating snapshot for volume 80"}
| 530 | 2 |
| org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd |
9 |
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
to create snapshot due to an internal error creating snapshot for volume 83"}
| 530 | 2 |
| org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd |
9 |
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
to create snapshot due to an internal error creating snapshot for volume 85"}
| 530 | 2 |
+-------------------------------------------------------------------+------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------+
4 rows in set (0.01 sec)
--
This message was sent by Atlassian JIRA
(v6.1#6144)
