With this log snippet what I could figure out is it issue with host (error
msg :callHostPlugin failed).clear host tag and try to force-reconnect it
should copy required packages .
On 5/30/16, 11:36 AM, "Venkat Boggarapu" <venka...@axiomio.com> wrote:
>Hi Glenn/Prashant,
>
>I am forwarding the exact log for the voloume snapshot
>Please check and let us know the error
>
>2016-05-29 01:07:10,021 WARN [xen.resource.CitrixResourceBase]
>(DirectAgent-250:null) Task failed! Task record: uuid:
>8f8e8e52-de8d-7310-22c5-030719540292
> nameLabel: Async.host.call_plugin
> nameDescription:
> allowedOperations: []
> currentOperations: {}
> created: Sun May 29 01:04:54 PDT 2016
> finished: Sun May 29 01:07:09 PDT 2016
> status: failure
> residentOn: com.xensource.xenapi.Host@2115a7db
> progress: 1.0
> type: <none/>
> result:
> errorInfo: [XENAPI_PLUGIN_FAILURE, backupSnapshot,
>CommandException, 32]
> otherConfig: {}
> subtaskOf: com.xensource.xenapi.Task@aaf13f6f
> subtasks: []
>
>2016-05-29 01:07:10,023 WARN [xen.resource.CitrixResourceBase]
>(DirectAgent-250:null) callHostPlugin failed for cmd: backupSnapshot with
>args secondaryStorageMountPath: 205.144.169.99:/export1/secondary,
>localMountPoint:
>/var/run/cloud_mount/60ef4a9f-de15-3c55-a95e-7c471608332c, backupUuid:
>6278b10c-3288-4e20-84b5-ada34b710e23, path: snapshots/3/1142,
>prevBackupUuid: snapshots/3/1142/a3585a4c-c2a1-4fe6-8fff-1ec799ecb7d6,
>snapshotUuid: 66e1b2cf-68bb-4245-ae0d-8a1fc6df18e3, isISCSI: true,
>primaryStorageSRUuid: 48be6dd0-cb35-132e-2661-99842fadbaba, due to Task
>failed! Task record: uuid:
>8f8e8e52-de8d-7310-22c5-030719540292
> nameLabel: Async.host.call_plugin
> nameDescription:
> allowedOperations: []
> currentOperations: {}
> created: Sun May 29 01:04:54 PDT 2016
> finished: Sun May 29 01:07:09 PDT 2016
> status: failure
> residentOn: com.xensource.xenapi.Host@2115a7db
> progress: 1.0
> type: <none/>
> result:
> errorInfo: [XENAPI_PLUGIN_FAILURE, backupSnapshot,
>CommandException, 32]
> otherConfig: {}
> subtaskOf: com.xensource.xenapi.Task@aaf13f6f
> subtasks: []
>
>Task failed! Task record: uuid:
>8f8e8e52-de8d-7310-22c5-030719540292
> nameLabel: Async.host.call_plugin
> nameDescription:
> allowedOperations: []
> currentOperations: {}
> created: Sun May 29 01:04:54 PDT 2016
> finished: Sun May 29 01:07:09 PDT 2016
> status: failure
> residentOn: com.xensource.xenapi.Host@2115a7db
> progress: 1.0
> type: <none/>
> result:
> errorInfo: [XENAPI_PLUGIN_FAILURE, backupSnapshot,
>CommandException, 32]
> otherConfig: {}
> subtaskOf: com.xensource.xenapi.Task@aaf13f6f
> subtasks: []
>
> at
>com.cloud.hypervisor.xen.resource.CitrixResourceBase.checkForSuccess(Citri
>xResourceBase.java:3605)
> at
>com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPluginAsync(C
>itrixResourceBase.java:3919)
> at
>com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.backupSnapshot
>(XenServerStorageProcessor.java:1117)
> at
>com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.backupSnapshot
>(XenServerStorageProcessor.java:1307)
> at
>com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(Stor
>ageSubsystemCommandHandlerBase.java:84)
> at
>com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorag
>eCommands(StorageSubsystemCommandHandlerBase.java:49)
> at
>com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(Citrix
>ResourceBase.java:621)
> at
>com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenSe
>rver56Resource.java:73)
> at
>com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenS
>erver610Resource.java:104)
> at
>com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.jav
>a:186)
> at
>java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at
>java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
>java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.acces
>s$101(ScheduledThreadPoolExecutor.java:165)
> at
>java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(S
>cheduledThreadPoolExecutor.java:266)
> at
>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
>1146)
> at
>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java
>:615)
> at java.lang.Thread.run(Thread.java:679)
>2016-05-29 01:07:11,040 WARN [xen.resource.XenServerStorageProcessor]
>(DirectAgent-250:null) Could not copy backupUuid: null from primary
>storage 48be6dd0-cb35-132e-2661-99842fadbaba to secondary storage
>205.144.169.99:/export1/secondary due to null
>2016-05-29 01:07:11,040 WARN [xen.resource.XenServerStorageProcessor]
>(DirectAgent-250:null) BackupSnapshot Failed due to Could not copy
>backupUuid: null from primary storage
>48be6dd0-cb35-132e-2661-99842fadbaba to secondary storage
>205.144.169.99:/export1/secondary due to null
>com.cloud.utils.exception.CloudRuntimeException: Could not copy
>backupUuid: null from primary storage
>48be6dd0-cb35-132e-2661-99842fadbaba to secondary storage
>205.144.169.99:/export1/secondary due to null
> at
>com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.backupSnapshot
>(XenServerStorageProcessor.java:1146)
> at
>com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.backupSnapshot
>(XenServerStorageProcessor.java:1307)
> at
>com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(Stor
>ageSubsystemCommandHandlerBase.java:84)
> at
>com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorag
>eCommands(StorageSubsystemCommandHandlerBase.java:49)
> at
>com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(Citrix
>ResourceBase.java:621)
> at
>com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenSe
>rver56Resource.java:73)
> at
>com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenSe
>rver56Resource.java:73)
> at
>com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenS
>erver610Resource.java:104)
> at
>com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.jav
>a:186)
> at
>java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at
>java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
>java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.acces
>s$101(ScheduledThreadPoolExecutor.java:165)
> at
>java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(S
>cheduledThreadPoolExecutor.java:266)
> at
>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
>1146)
> at
>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java
>:615)
> at java.lang.Thread.run(Thread.java:679)
>2016-05-29 01:07:11,041 DEBUG [agent.manager.DirectAgentAttache]
>(DirectAgent-250:null) Seq 39-2062304528: Response Received:
>2016-05-29 01:07:11,041 DEBUG [agent.transport.Request]
>(DirectAgent-250:null) Seq 39-2062304528: Processing: { Ans: , MgmtId:
>120860783173, via: 39, Ver: v1, Flags: 110,
>[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"d
>etails":"BackupSnapshot Failed due to Could not copy backupUuid: null
>from primary storage 48be6dd0-cb35-132e-2661-99842fadbaba to secondary
>storage 205.144.169.99:/export1/secondary due to null","wait":0}}] }
>2016-05-29 01:07:11,042 DEBUG [agent.manager.AgentAttache]
>(DirectAgent-250:null) Seq 39-2062304541: Sending now. is current
>sequence.
>2016-05-29 01:07:11,042 DEBUG [agent.transport.Request]
>(Job-Executor-10:job-31466 = [ f3aca400-b871-41b9-927d-18b1bd8b0635 ])
>Seq 39-2062304528: Received: { Ans: , MgmtId: 120860783173, via: 39,
>Ver: v1, Flags: 110, { CopyCmdAnswer } }
>2016-05-29 01:07:11,042 DEBUG [agent.transport.Request]
>(DirectAgent-250:null) Seq 39-2062304541: Executing: { Cmd , MgmtId:
>120860783173, via: 39, Ver: v1, Flags: 100111,
>[{"com.cloud.agent.api.CheckS2SVpnConnectionsCommand":{"vpnIps":[],"access
>Details":{"router.ip":"169.254.1.111","router.name":"r-781-VM"},"wait":30}
>}] }
>2016-05-29 01:07:11,042 DEBUG [agent.manager.DirectAgentAttache]
>(DirectAgent-250:null) Seq 39-2062304541: Executing request
>2016-05-29 01:07:11,056 DEBUG [storage.snapshot.SnapshotManagerImpl]
>(Job-Executor-10:job-31466 = [ f3aca400-b871-41b9-927d-18b1bd8b0635 ])
>Failed to create snapshot
>com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot Failed
>due to Could not copy backupUuid: null from primary storage
>48be6dd0-cb35-132e-2661-99842fadbaba to secondary storage
>205.144.169.99:/export1/secondary due to null
> at
>org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(
>SnapshotServiceImpl.java:280)
> at
>org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSna
>pshot(XenserverSnapshotStrategy.java:138)
> at
>org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnaps
>hot(XenserverSnapshotStrategy.java:264)
> at
>com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManage
>rImpl.java:1013)
> at
>com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorD
>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
>org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(Volume
>ServiceImpl.java:1307)
> at
>com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:27
>19)
> at
>org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(
>CreateSnapshotCmd.java:170)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> at
>com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> at
>java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at
>java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
>1146)
> at
>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java
>:615)
> at java.lang.Thread.run(Thread.java:679)
>2016-05-29 01:07:11,063 DEBUG [storage.volume.VolumeServiceImpl]
>(Job-Executor-10:job-31466 = [ f3aca400-b871-41b9-927d-18b1bd8b0635 ])
>Take snapshot: 1142 failed
>com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot
> at
>com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManage
>rImpl.java:1040)
> at
>com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:27
>19)
> at
>org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(
>CreateSnapshotCmd.java:170)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> at
>com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> at
>java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at
>java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
>1146)
> at
>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java
>:615)
> at java.lang.Thread.run(Thread.java:679)
>Caused by: com.cloud.utils.exception.CloudRuntimeException:
>BackupSnapshot Failed due to Could not copy backupUuid: null from primary
>storage 48be6dd0-cb35-132e-2661-99842fadbaba to secondary storage
>205.144.169.99:/export1/secondary due to null
> at
>org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(
>SnapshotServiceImpl.java:280)
> at
>org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSna
>pshot(XenserverSnapshotStrategy.java:138)
> at
>org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnaps
>hot(XenserverSnapshotStrategy.java:264)
> at
>com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManage
>rImpl.java:1013)
> ... 16 more
>2016-05-29 01:07:11,070 DEBUG [cloud.async.AsyncJobManagerImpl]
>(Job-Executor-10:job-31466 = [ f3aca400-b871-41b9-927d-18b1bd8b0635 ])
>Complete async job-31466 = [ f3aca400-b871-41b9-927d-18b1bd8b0635 ],
>jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text: Failed
>to create snapshot due to an internal error creating snapshot for volume
>1142
>
>-----Original Message-----
>From: Prashant Mishra [mailto:prashant.mis...@accelerite.com]
>Sent: Friday, May 27, 2016 10:06 AM
>To: users@cloudstack.apache.org
>Subject: Re: DAILY SNAPSHOT ISSUE FOR VOLUME SNAPSHOT
>
>As Glenn said please provide MS log for further investigation , CREATED
>ON PRIMARY is the state of snapshot ?
>On 5/27/16, 9:57 AM, "Venkat Boggarapu" <venka...@axiomio.com> wrote:
>
>>Hi Glenn,
>>
>>I tried to remove the error and taken manual snapshot for the volume,
>>manual snapshot is done, after that for two days daily snapshots is
>>working good for that volume.
>>
>>After two days the error comes again as "CREATED ON PRIMARY". This is
>>happening from last three to four weeks.
>>
>>Thanks,
>>Venkat Boggarapu
>>
>>-----Original Message-----
>>From: Glenn Wagner [mailto:glenn.wag...@shapeblue.com]
>>Sent: Thursday, May 26, 2016 1:04 PM
>>To: users@cloudstack.apache.org
>>Subject: RE: DAILY SNAPSHOT ISSUE FOR VOLUME SNAPSHOT
>>
>>Hi,
>>
>>Can you upload (pastebin) your management server logs for us to see the
>>error,
>>
>>Regards
>>Glenn
>>
>>glenn.wag...@shapeblue.com
>>www.shapeblue.com
>>2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West, Cape Town
>>7130South Africa @shapeblue
>>
>>
>>
>>-----Original Message-----
>>From: Venkat Boggarapu [mailto:venka...@axiomio.com]
>>Sent: Thursday, 26 May 2016 9:29 AM
>>To: users@cloudstack.apache.org
>>Subject: DAILY SNAPSHOT ISSUE FOR VOLUME SNAPSHOT
>>
>>Hi Guys,
>>
>>My environment is Cloudstack-4.2.0 and Xenserver is 6.2.0
>>
>>I kept daily scheduled volume snapshots for all the vms in my
>>environment.
>>
>>But from last three weeks I am getting error "CREATED ON PRIMARY" for
>>one volume snapshot on ROOT volume.
>>
>>Please help me regarding if any one faced the same issue.
>>
>>Thanks,
>>Venkat Boggarapu
>
>
>
>
>DISCLAIMER
>==========
>This e-mail may contain privileged and confidential information which is
>the property of Accelerite, a Persistent Systems business. It is intended
>only for the use of the individual or entity to which it is addressed. If
>you are not the intended recipient, you are not authorized to read,
>retain, copy, print, distribute or use this message. If you have received
>this communication in error, please notify the sender and delete all
>copies of this message. Accelerite, a Persistent Systems business does
>not accept any liability for virus infected mails.
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the
property of Accelerite, a Persistent Systems business. It is intended only for
the use of the individual or entity to which it is addressed. If you are not
the intended recipient, you are not authorized to read, retain, copy, print,
distribute or use this message. If you have received this communication in
error, please notify the sender and delete all copies of this message.
Accelerite, a Persistent Systems business does not accept any liability for
virus infected mails.
