The OS is CentOS 6.5, Storage is NFS not hosted on the controller. I will also
note that there is two zones, with the same configuration just using different
NFS devices one zone can take snapshots but the one that can no longer take
them has been working for 8 months with no issues.
Here is the section from the log file when I try to take a snapshot:
2014-11-26 07:00:48,308 DEBUG [cloud.api.ApiServlet] (catalina-exec-4:null)
===START=== 172.16.28.20 -- GET
command=createSnapshot&volumeid=ec6460c4-896c-46d1-8ae5-75d6431a6608&response=json&sessionkey=ysbTjq7whHzoePtvcRBRSzsJ6zw%3D&_=1417014045703
2014-11-26 07:00:48,362 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-4:null) submit async job-147 = [
2007bb4e-03c9-4ff7-874d-e50641318ace ], details: AsyncJobVO {id:147, userId: 2,
accountId: 2, sessionKey: null, instanceType: Snapshot, instanceId: 128, cmd:
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd,
cmdOriginator: null, cmdInfo:
{"id":"128","response":"json","sessionkey":"ysbTjq7whHzoePtvcRBRSzsJ6zw\u003d","cmdEventType":"SNAPSHOT.CREATE","ctxUserId":"2","httpmethod":"GET","_":"1417014045703","volumeid":"ec6460c4-896c-46d1-8ae5-75d6431a6608","ctxAccountId":"2","ctxStartEventId":"5425"},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
processStatus: 0, resultCode: 0, result: null, initMsid: 345048644703,
completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2014-11-26 07:00:48,363 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-65:job-147 = [ 2007bb4e-03c9-4ff7-874d-e50641318ace ]) Executing
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-147 =
[ 2007bb4e-03c9-4ff7-874d-e50641318ace ]
2014-11-26 07:00:48,363 DEBUG [cloud.api.ApiServlet] (catalina-exec-4:null)
===END=== 172.16.28.20 -- GET
command=createSnapshot&volumeid=ec6460c4-896c-46d1-8ae5-75d6431a6608&response=json&sessionkey=ysbTjq7whHzoePtvcRBRSzsJ6zw%3D&_=1417014045703
2014-11-26 07:00:48,386 INFO [user.snapshot.CreateSnapshotCmd]
(Job-Executor-65:job-147 = [ 2007bb4e-03c9-4ff7-874d-e50641318ace ]) VOLSS:
createSnapshotCmd starts:1417014048386
2014-11-26 07:00:48,437 DEBUG [agent.transport.Request]
(Job-Executor-65:job-147 = [ 2007bb4e-03c9-4ff7-874d-e50641318ace ]) Seq
7-1985284483: Sending { Cmd , MgmtId: 345048644703, via: 7, Ver: v1, Flags:
100011,
[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"ec6460c4-896c-46d1-8ae5-75d6431a6608","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"055b61d2-5260-3856-92fc-48facf5dfbaf","id":2,"poolType":"VMFS","host":"VMFS
datastore:
/Labs/CS-PRI-02","path":"/Labs/CS-PRI-02","port":0}},"name":"ROOT-297","size":21474836480,"path":"ROOT-297-000001","volumeId":334,"vmName":"i-2-297-VM","accountId":2,"chainInfo":"{\"diskDeviceBusName\":\"ide0:1\",\"diskChain\":[\"[CS-PRI-02]
i-2-297-VM/ROOT-297.vmdk\"]}","format":"OVA","id":334,"hypervisorType":"VMware"},"parentSnapshotPath":"2e7af1a5-a31c-4fe6-b2f7-21965c1a86b9","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"055b61d2-5260-3856-92fc-48facf5dfbaf","id":2,"poolType":"VMFS","host":"VMFS
datastore:
/Labs/CS-PRI-02","path":"/Labs/CS-PRI-02","port":0}},"vmName":"i-2-297-VM","name":"new-import_ROOT-297_20141126150048","hypervisorType":"VMware","id":128}},"wait":0}}]
}
2014-11-26 07:00:48,438 DEBUG [agent.transport.Request]
(Job-Executor-65:job-147 = [ 2007bb4e-03c9-4ff7-874d-e50641318ace ]) Seq
7-1985284483: Executing: { Cmd , MgmtId: 345048644703, via: 7, Ver: v1, Flags:
100011,
[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"ec6460c4-896c-46d1-8ae5-75d6431a6608","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"055b61d2-5260-3856-92fc-48facf5dfbaf","id":2,"poolType":"VMFS","host":"VMFS
datastore:
/Labs/CS-PRI-02","path":"/Labs/CS-PRI-02","port":0}},"name":"ROOT-297","size":21474836480,"path":"ROOT-297-000001","volumeId":334,"vmName":"i-2-297-VM","accountId":2,"chainInfo":"{\"diskDeviceBusName\":\"ide0:1\",\"diskChain\":[\"[CS-PRI-02]
i-2-297-VM/ROOT-297.vmdk\"]}","format":"OVA","id":334,"hypervisorType":"VMware"},"parentSnapshotPath":"2e7af1a5-a31c-4fe6-b2f7-21965c1a86b9","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"055b61d2-5260-3856-92fc-48facf5dfbaf","id":2,"poolType":"VMFS","host":"VMFS
datastore:
/Labs/CS-PRI-02","path":"/Labs/CS-PRI-02","port":0}},"vmName":"i-2-297-VM","name":"new-import_ROOT-297_20141126150048","hypervisorType":"VMware","id":128}},"wait":0}}]
}
2014-11-26 07:00:48,438 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-311:null) Seq 7-1985284483: Executing request
2014-11-26 07:00:48,438 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-311:null) Seq 7-1985284483: Response Received:
2014-11-26 07:00:48,438 DEBUG [agent.transport.Request] (DirectAgent-311:null)
Seq 7-1985284483: Processing: { Ans: , MgmtId: 345048644703, via: 7, Ver: v1,
Flags: 10,
[{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"b7c08601-0654-4749-b7e0-ccc380458e3d","id":0}},"result":true,"wait":0}}]
}
2014-11-26 07:00:48,438 DEBUG [agent.transport.Request]
(Job-Executor-65:job-147 = [ 2007bb4e-03c9-4ff7-874d-e50641318ace ]) Seq
7-1985284483: Received: { Ans: , MgmtId: 345048644703, via: 7, Ver: v1, Flags:
10, { CreateObjectAnswer } }
2014-11-26 07:00:48,488 DEBUG [storage.motion.AncientDataMotionStrategy]
(Job-Executor-65:job-147 = [ 2007bb4e-03c9-4ff7-874d-e50641318ace ]) copyAsync
inspecting src type SNAPSHOT copyAsync inspecting dest type SNAPSHOT
2014-11-26 07:00:48,535 DEBUG [agent.transport.Request]
(Job-Executor-65:job-147 = [ 2007bb4e-03c9-4ff7-874d-e50641318ace ]) Seq
9-1742209940: Sending { Cmd , MgmtId: 345048644703, via: 9, Ver: v1, Flags:
100111,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"b7c08601-0654-4749-b7e0-ccc380458e3d","volume":{"uuid":"ec6460c4-896c-46d1-8ae5-75d6431a6608","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"055b61d2-5260-3856-92fc-48facf5dfbaf","id":2,"poolType":"VMFS","host":"VMFS
datastore:
/Labs/CS-PRI-02","path":"/Labs/CS-PRI-02","port":0}},"name":"ROOT-297","size":21474836480,"path":"ROOT-297-000001","volumeId":334,"vmName":"i-2-297-VM","accountId":2,"chainInfo":"{\"diskDeviceBusName\":\"ide0:1\",\"diskChain\":[\"[CS-PRI-02]
i-2-297-VM/ROOT-297.vmdk\"]}","format":"OVA","id":334,"hypervisorType":"VMware"},"parentSnapshotPath":"2e7af1a5-a31c-4fe6-b2f7-21965c1a86b9","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"055b61d2-5260-3856-92fc-48facf5dfbaf","id":2,"poolType":"VMFS","host":"VMFS
datastore:
/Labs/CS-PRI-02","path":"/Labs/CS-PRI-02","port":0}},"vmName":"i-2-297-VM","name":"new-import_ROOT-297_20141126150048","hypervisorType":"VMware","id":128}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/334","volume":{"uuid":"ec6460c4-896c-46d1-8ae5-75d6431a6608","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"055b61d2-5260-3856-92fc-48facf5dfbaf","id":2,"poolType":"VMFS","host":"VMFS
datastore:
/Labs/CS-PRI-02","path":"/Labs/CS-PRI-02","port":0}},"name":"ROOT-297","size":21474836480,"path":"ROOT-297-000001","volumeId":334,"vmName":"i-2-297-VM","accountId":2,"chainInfo":"{\"diskDeviceBusName\":\"ide0:1\",\"diskChain\":[\"[CS-PRI-02]
i-2-297-VM/ROOT-297.vmdk\"]}","format":"OVA","id":334,"hypervisorType":"VMware"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://172.16.25.84/data/CloudStackSEC2","_role":"Image"}},"vmName":"i-2-297-VM","name":"new-import_ROOT-297_20141126150048","hypervisorType":"VMware","id":128}},"executeInSequence":true,"wait":21600}}]
}
2014-11-26 07:00:51,404 DEBUG [cloud.api.ApiServlet] (catalina-exec-25:null)
===START=== 172.16.28.20 -- GET
command=queryAsyncJobResult&jobId=2007bb4e-03c9-4ff7-874d-e50641318ace&response=json&sessionkey=ysbTjq7whHzoePtvcRBRSzsJ6zw%3D&_=1417014048864
2014-11-26 07:00:51,428 DEBUG [cloud.api.ApiServlet] (catalina-exec-25:null)
===END=== 172.16.28.20 -- GET
command=queryAsyncJobResult&jobId=2007bb4e-03c9-4ff7-874d-e50641318ace&response=json&sessionkey=ysbTjq7whHzoePtvcRBRSzsJ6zw%3D&_=1417014048864
2014-11-26 07:00:51,840 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-2:null) SeqA 3-1302876: Processing Seq 3-1302876: { Cmd
, MgmtId: -1, via: 3, Ver: v1, Flags: 11,
[{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":2,"_loadInfo":"{\n
\"connections\": []\n}","wait":0}}] }
2014-11-26 07:00:51,846 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-2:null) SeqA 3-1302876: Sending Seq 3-1302876: { Ans: ,
MgmtId: 345048644703, via: 3, Ver: v1, Flags: 100010,
[{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
2014-11-26 07:00:52,150 DEBUG [agent.transport.Request]
(AgentManager-Handler-15:null) Seq 9-1742209940: Processing: { Ans: , MgmtId:
345048644703, via: 9, Ver: v1, Flags: 110,
[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"backup
snapshot exception: Exception: java.lang.Exception\nMessage: unable to prepare
snapshot backup directory\n","wait":0}}] }
2014-11-26 07:00:52,150 DEBUG [agent.transport.Request]
(Job-Executor-65:job-147 = [ 2007bb4e-03c9-4ff7-874d-e50641318ace ]) Seq
9-1742209940: Received: { Ans: , MgmtId: 345048644703, via: 9, Ver: v1, Flags:
110, { CopyCmdAnswer } }
2014-11-26 07:00:52,155 DEBUG [agent.manager.AgentAttache]
(AgentManager-Handler-15:null) Seq 9-1742209940: No more commands found
2014-11-26 07:00:52,171 DEBUG [storage.snapshot.SnapshotManagerImpl]
(Job-Executor-65:job-147 = [ 2007bb4e-03c9-4ff7-874d-e50641318ace ]) Failed to
create snapshot
com.cloud.utils.exception.CloudRuntimeException: backup snapshot exception:
Exception: java.lang.Exception
Message: unable to prepare snapshot backup directory
at
org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280)
at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:136)
at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:279)
at
com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1007)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1311)
at
com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2773)
at
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-11-26 07:00:52,184 DEBUG [storage.volume.VolumeServiceImpl]
(Job-Executor-65:job-147 = [ 2007bb4e-03c9-4ff7-874d-e50641318ace ]) Take
snapshot: 334 failed
com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot
at
com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1034)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1311)
at
com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2773)
at
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.cloud.utils.exception.CloudRuntimeException: backup snapshot
exception: Exception: java.lang.Exception
Message: unable to prepare snapshot backup directory
at
org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280)
at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:136)
at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:279)
at
com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1007)
... 15 more
2014-11-26 07:00:52,184 DEBUG [user.snapshot.CreateSnapshotCmd]
(Job-Executor-65:job-147 = [ 2007bb4e-03c9-4ff7-874d-e50641318ace ]) Failed to
create snapshot
org.apache.cloudstack.api.ServerApiException
at
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:176)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
-----Original Message-----
From: Andrija Panic [mailto:[email protected]]
Sent: Tuesday, November 25, 2014 11:55 PM
To: [email protected]
Subject: Re: Unable to take Snapshot CS 4.2.1
Please attach the management logs for further invetigation...what OS, what
storage, etc...
On 26 November 2014 at 01:51, Jeff Barnett <[email protected]> wrote:
> When I try to take a snapshot now I just get an error like this,
> Failed to create a snapshot due to an internal error creating snapshot for
> volume ...
> has anyone run into this, if so what was the fix, I tried restarting
> the cloud services on the SSVM, but that did not help.
>
>
>
>
--
Andrija Panić
