Hi Glenn/Prashant, I am forwarding the exact log for the voloume snapshot Please check and let us know the error
2016-05-29 01:07:10,021 WARN [xen.resource.CitrixResourceBase] (DirectAgent-250:null) Task failed! Task record: uuid: 8f8e8e52-de8d-7310-22c5-030719540292 nameLabel: Async.host.call_plugin nameDescription: allowedOperations: [] currentOperations: {} created: Sun May 29 01:04:54 PDT 2016 finished: Sun May 29 01:07:09 PDT 2016 status: failure residentOn: com.xensource.xenapi.Host@2115a7db progress: 1.0 type: <none/> result: errorInfo: [XENAPI_PLUGIN_FAILURE, backupSnapshot, CommandException, 32] otherConfig: {} subtaskOf: com.xensource.xenapi.Task@aaf13f6f subtasks: [] 2016-05-29 01:07:10,023 WARN [xen.resource.CitrixResourceBase] (DirectAgent-250:null) callHostPlugin failed for cmd: backupSnapshot with args secondaryStorageMountPath: 205.144.169.99:/export1/secondary, localMountPoint: /var/run/cloud_mount/60ef4a9f-de15-3c55-a95e-7c471608332c, backupUuid: 6278b10c-3288-4e20-84b5-ada34b710e23, path: snapshots/3/1142, prevBackupUuid: snapshots/3/1142/a3585a4c-c2a1-4fe6-8fff-1ec799ecb7d6, snapshotUuid: 66e1b2cf-68bb-4245-ae0d-8a1fc6df18e3, isISCSI: true, primaryStorageSRUuid: 48be6dd0-cb35-132e-2661-99842fadbaba, due to Task failed! Task record: uuid: 8f8e8e52-de8d-7310-22c5-030719540292 nameLabel: Async.host.call_plugin nameDescription: allowedOperations: [] currentOperations: {} created: Sun May 29 01:04:54 PDT 2016 finished: Sun May 29 01:07:09 PDT 2016 status: failure residentOn: com.xensource.xenapi.Host@2115a7db progress: 1.0 type: <none/> result: errorInfo: [XENAPI_PLUGIN_FAILURE, backupSnapshot, CommandException, 32] otherConfig: {} subtaskOf: com.xensource.xenapi.Task@aaf13f6f subtasks: [] Task failed! Task record: uuid: 8f8e8e52-de8d-7310-22c5-030719540292 nameLabel: Async.host.call_plugin nameDescription: allowedOperations: [] currentOperations: {} created: Sun May 29 01:04:54 PDT 2016 finished: Sun May 29 01:07:09 PDT 2016 status: failure residentOn: com.xensource.xenapi.Host@2115a7db progress: 1.0 type: <none/> result: errorInfo: [XENAPI_PLUGIN_FAILURE, backupSnapshot, CommandException, 32] otherConfig: {} subtaskOf: com.xensource.xenapi.Task@aaf13f6f subtasks: [] at com.cloud.hypervisor.xen.resource.CitrixResourceBase.checkForSuccess(CitrixResourceBase.java:3605) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPluginAsync(CitrixResourceBase.java:3919) at com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.backupSnapshot(XenServerStorageProcessor.java:1117) at com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.backupSnapshot(XenServerStorageProcessor.java:1307) at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:84) at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:621) at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:73) at com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:104) at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:679) 2016-05-29 01:07:11,040 WARN [xen.resource.XenServerStorageProcessor] (DirectAgent-250:null) Could not copy backupUuid: null from primary storage 48be6dd0-cb35-132e-2661-99842fadbaba to secondary storage 205.144.169.99:/export1/secondary due to null 2016-05-29 01:07:11,040 WARN [xen.resource.XenServerStorageProcessor] (DirectAgent-250:null) BackupSnapshot Failed due to Could not copy backupUuid: null from primary storage 48be6dd0-cb35-132e-2661-99842fadbaba to secondary storage 205.144.169.99:/export1/secondary due to null com.cloud.utils.exception.CloudRuntimeException: Could not copy backupUuid: null from primary storage 48be6dd0-cb35-132e-2661-99842fadbaba to secondary storage 205.144.169.99:/export1/secondary due to null at com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.backupSnapshot(XenServerStorageProcessor.java:1146) at com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.backupSnapshot(XenServerStorageProcessor.java:1307) at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:84) at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:621) at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:73) at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:73) at com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:104) at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:679) 2016-05-29 01:07:11,041 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-250:null) Seq 39-2062304528: Response Received: 2016-05-29 01:07:11,041 DEBUG [agent.transport.Request] (DirectAgent-250:null) Seq 39-2062304528: Processing: { Ans: , MgmtId: 120860783173, via: 39, Ver: v1, Flags: 110, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"BackupSnapshot Failed due to Could not copy backupUuid: null from primary storage 48be6dd0-cb35-132e-2661-99842fadbaba to secondary storage 205.144.169.99:/export1/secondary due to null","wait":0}}] } 2016-05-29 01:07:11,042 DEBUG [agent.manager.AgentAttache] (DirectAgent-250:null) Seq 39-2062304541: Sending now. is current sequence. 2016-05-29 01:07:11,042 DEBUG [agent.transport.Request] (Job-Executor-10:job-31466 = [ f3aca400-b871-41b9-927d-18b1bd8b0635 ]) Seq 39-2062304528: Received: { Ans: , MgmtId: 120860783173, via: 39, Ver: v1, Flags: 110, { CopyCmdAnswer } } 2016-05-29 01:07:11,042 DEBUG [agent.transport.Request] (DirectAgent-250:null) Seq 39-2062304541: Executing: { Cmd , MgmtId: 120860783173, via: 39, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.CheckS2SVpnConnectionsCommand":{"vpnIps":[],"accessDetails":{"router.ip":"169.254.1.111","router.name":"r-781-VM"},"wait":30}}] } 2016-05-29 01:07:11,042 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-250:null) Seq 39-2062304541: Executing request 2016-05-29 01:07:11,056 DEBUG [storage.snapshot.SnapshotManagerImpl] (Job-Executor-10:job-31466 = [ f3aca400-b871-41b9-927d-18b1bd8b0635 ]) Failed to create snapshot com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot Failed due to Could not copy backupUuid: null from primary storage 48be6dd0-cb35-132e-2661-99842fadbaba to secondary storage 205.144.169.99:/export1/secondary due to null at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:138) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:264) at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1013) at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1307) at com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2719) at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158) at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:679) 2016-05-29 01:07:11,063 DEBUG [storage.volume.VolumeServiceImpl] (Job-Executor-10:job-31466 = [ f3aca400-b871-41b9-927d-18b1bd8b0635 ]) Take snapshot: 1142 failed com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1040) at com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2719) at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158) at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:679) Caused by: com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot Failed due to Could not copy backupUuid: null from primary storage 48be6dd0-cb35-132e-2661-99842fadbaba to secondary storage 205.144.169.99:/export1/secondary due to null at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:138) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:264) at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1013) ... 16 more 2016-05-29 01:07:11,070 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-10:job-31466 = [ f3aca400-b871-41b9-927d-18b1bd8b0635 ]) Complete async job-31466 = [ f3aca400-b871-41b9-927d-18b1bd8b0635 ], jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text: Failed to create snapshot due to an internal error creating snapshot for volume 1142 -----Original Message----- From: Prashant Mishra [mailto:prashant.mis...@accelerite.com] Sent: Friday, May 27, 2016 10:06 AM To: users@cloudstack.apache.org Subject: Re: DAILY SNAPSHOT ISSUE FOR VOLUME SNAPSHOT As Glenn said please provide MS log for further investigation , CREATED ON PRIMARY is the state of snapshot ? On 5/27/16, 9:57 AM, "Venkat Boggarapu" <venka...@axiomio.com> wrote: >Hi Glenn, > >I tried to remove the error and taken manual snapshot for the volume, >manual snapshot is done, after that for two days daily snapshots is >working good for that volume. > >After two days the error comes again as "CREATED ON PRIMARY". This is >happening from last three to four weeks. > >Thanks, >Venkat Boggarapu > >-----Original Message----- >From: Glenn Wagner [mailto:glenn.wag...@shapeblue.com] >Sent: Thursday, May 26, 2016 1:04 PM >To: users@cloudstack.apache.org >Subject: RE: DAILY SNAPSHOT ISSUE FOR VOLUME SNAPSHOT > >Hi, > >Can you upload (pastebin) your management server logs for us to see the >error, > >Regards >Glenn > >glenn.wag...@shapeblue.com >www.shapeblue.com >2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West, Cape Town >7130South Africa @shapeblue > > > >-----Original Message----- >From: Venkat Boggarapu [mailto:venka...@axiomio.com] >Sent: Thursday, 26 May 2016 9:29 AM >To: users@cloudstack.apache.org >Subject: DAILY SNAPSHOT ISSUE FOR VOLUME SNAPSHOT > >Hi Guys, > >My environment is Cloudstack-4.2.0 and Xenserver is 6.2.0 > >I kept daily scheduled volume snapshots for all the vms in my environment. > >But from last three weeks I am getting error "CREATED ON PRIMARY" for >one volume snapshot on ROOT volume. > >Please help me regarding if any one faced the same issue. > >Thanks, >Venkat Boggarapu DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.