Sangeetha Hariharan created CLOUDSTACK-4734:
-----------------------------------------------
Summary: Creating snapshot from ROOT volume fails with error
message - "Failed to create snapshot due to an internal error creating snapshot
for volume 14”
Key: CLOUDSTACK-4734
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4734
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Management Server
Affects Versions: 4.2.1
Environment: Build from 4.2-forward
Reporter: Sangeetha Hariharan
Fix For: 4.2.1
Set up:
3.0.7 Advanced Zone setup with 2 Zones having Vmware EXSI 5.0 clusters on
Vcenter 5.0 .
Upgraded to 4.2.0.
Tried to take snapshot for ROOT volume of 2 existing Vms from 2 different zones.
Snapshots got created successfully.
After few hours , when I tried to create snapshot from ROOT volume for a VM, I
was not able to create snapshots from ROOT volume.
I get the following error – “Error Code: 530 Error text: Failed to create
snapshot due to an internal error creating snapshot for volume
14”
Following is the error message seen on management server logs:
2013-09-21 06:56:13,573 DEBUG [agent.transport.Request] (Job-Executor-36:job-88
= [ c7cbee54-930d-4020-ae02-4710806
bdc00 ]) Seq 9-1758069619: Sending { Cmd , MgmtId: 161197867246747, via: 9,
Ver: v1, Flags: 100111, [{"org.apache.
cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"9e8a
10cf-7cbc-4bef-a57b-563edb6b62ed","volume":{"uuid":"05a6bdfd-e6a5-4ae5-9dab-d06ee40776e9","volumeType":"ROOT","data
Store":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"61e1621f-a1a8-336d-b18d-050586f19b09","id":2
00,"poolType":"NetworkFilesystem","host":"10.223.110.232","path":"/export/home/sangeetha/campofinalga/primary","por
t":2049}},"name":"ROOT-14","size":2147483648,"path":"ROOT-14-14","volumeId":14,"vmName":"i-3-14-VM","accountId":3,"
format":"OVA","id":14,"hypervisorType":"VMware"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO"
:{"uuid":"61e1621f-a1a8-336d-b18d-050586f19b09","id":200,"poolType":"NetworkFilesystem","host":"10.223.110.232","pa
th":"/export/home/sangeetha/campofinalga/primary","port":2049}},"vmName":"i-3-14-VM","name":"sangee-tier1-4_ROOT-14
_20130921135613","hypervisorType":"VMware","id":5}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{
"path":"snapshots/3/14","volume":{"uuid":"05a6bdfd-e6a5-4ae5-9dab-d06ee40776e9","volumeType":"ROOT","dataStore":{"o
rg.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"61e1621f-a1a8-336d-b18d-050586f19b09","id":200,"poolTy
pe":"NetworkFilesystem","host":"10.223.110.232","path":"/export/home/sangeetha/campofinalga/primary","port":2049}},
"name":"ROOT-14","size":2147483648,"path":"ROOT-14-14","volumeId":14,"vmName":"i-3-14-VM","accountId":3,"format":"O
VA","id":14,"hypervisorType":"VMware"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.223.110.232/ex
port/home/sangeetha/campofinalga/secondary","_role":"Image"}},"vmName":"i-3-14-VM","name":"sangee-tier1-4_ROOT-14_2
0130921135613","hypervisorType":"VMware","id":5}},"executeInSequence":true,"wait":21600}}]
}
2013-09-21 06:56:18,263 DEBUG [agent.transport.Request]
(AgentManager-Handler-14:null) Seq 9-1758069619: Processing
: { Ans: , MgmtId: 161197867246747, via: 9, Ver: v1, Flags: 110,
[{"org.apache.cloudstack.storage.command.CopyCmdA
nswer":{"result":false,"details":"backup snapshot exception: Exception:
javax.xml.ws.soap.SOAPFaultException\nMessa
ge: \nRequired property obj is missing from data object of type
ObjectSpec\n\nwhile parsing serialized DataObject o
f type vmodl.query.PropertyCollector.ObjectSpec\nat line 1, column 305\n\nwhile
parsing property \"objectSet\" of s
tatic type ArrayOfObjectSpec\n\nwhile parsing serialized DataObject of type
vmodl.query.PropertyCollector.FilterSpe
c\nat line 1, column 205\n\nwhile parsing call information for method
RetrieveProperties\nat line 1, column 110\n\n
while parsing SOAP body\nat line 1, column 102\n\nwhile parsing SOAP
envelope\nat line 1, column 38\n\nwhile parsin
g HTTP request for method retrieveContents\non object of type
vmodl.query.PropertyCollector\nat line 1, column 0\n"
,"wait":0}}] }
2013-09-21 06:56:18,264 DEBUG [agent.transport.Request] (Job-Executor-36:job-88
= [ c7cbee54-930d-4020-ae02-4710806
bdc00 ]) Seq 9-1758069619: Received: { Ans: , MgmtId: 161197867246747, via: 9,
Ver: v1, Flags: 110, { CopyCmdAnswe
r } }
2013-09-21 06:56:18,270 DEBUG [agent.manager.AgentAttache]
(AgentManager-Handler-14:null) Seq 9-1758069619: No more
commands found
2013-09-21 06:56:18,291 DEBUG [storage.snapshot.SnapshotManagerImpl]
(Job-Executor-36:job-88 = [ c7cbee54-930d-4020
-ae02-4710806bdc00 ]) Failed to create snapshot
com.cloud.utils.exception.CloudRuntimeException: backup snapshot exception:
Exception: javax.xml.ws.soap.SOAPFaultE
xception
Message:
Required property obj is missing from data object of type ObjectSpec
while parsing serialized DataObject of type
vmodl.query.PropertyCollector.ObjectSpec
at line 1, column 305
while parsing property "objectSet" of static type ArrayOfObjectSpec
while parsing serialized DataObject of type
vmodl.query.PropertyCollector.FilterSpec
at line 1, column 205
while parsing call information for method RetrieveProperties
at line 1, column 110
while parsing SOAP body
at line 1, column 102
while parsing SOAP envelope
at line 1, column 38
while parsing HTTP request for method retrieveContents
on object of type vmodl.query.PropertyCollector
at line 1, column 0
at
org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280)
at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrateg
y.java:139)
at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.
java:277)
at
com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1013)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentI
nstantiationPostProcessor.java:125)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1302)
at
com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2739)
at
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-09-21 06:56:18,299 DEBUG [storage.volume.VolumeServiceImpl]
(Job-Executor-36:job-88 = [ c7cbee54-930d-4020-ae0
2-4710806bdc00 ]) Take snapshot: 14 failed
com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot
at
com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1040)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentI
nstantiationPostProcessor.java:125)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1302)
at
com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2739)
at
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: com.cloud.utils.exception.CloudRuntimeException: backup snapshot
exception: Exception: javax.xml.ws.soap
.SOAPFaultException
Message:
Required property obj is missing from data object of type ObjectSpec
while parsing serialized DataObject of type
vmodl.query.PropertyCollector.ObjectSpec
at line 1, column 305
while parsing property "objectSet" of static type ArrayOfObjectSpec
while parsing serialized DataObject of type
vmodl.query.PropertyCollector.FilterSpec
at line 1, column 205
while parsing call information for method RetrieveProperties
at line 1, column 110
while parsing SOAP body
at line 1, column 102
while parsing SOAP envelope
at line 1, column 38
while parsing HTTP request for method retrieveContents
on object of type vmodl.query.PropertyCollector
at line 1, column 0
at
org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280)
at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:139)
at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:277)
at
com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1013)
... 16 more
2013-09-21 06:56:18,300 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-36:job-88 = [ c7cbee54-930d-4020-ae02-4710806bdc00 ]) Complete
async job-88 = [ c7cbee54-930d-4020-ae02-4710806bdc00 ], jobStatus: 2,
resultCode: 530, result: Error Code: 530 Error text: Failed to create snapshot
due to an internal error creating snapshot for volume 14
2013-09-21 06:56:20,404 DEBUG [cloud.api.ApiServlet] (catalina-exec-14:null)
===START=== 10.216.51.147 -- GET
command=queryAsyncJobResult&jobId=c7cbee54-930d-4020-ae02-4710806bdc00&response=json&sessionkey=lJPNS2cs%2FvyeTKM2Si1gP8tBG1w%3D&_=1379772417612
2013-09-21 06:56:20,413 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-14:null) Async job-88 = [ c7cbee54-930d-4020-ae02-4710806bdc00 ]
completed
In the logs I see the following exceptions from ~9:15 yesterday for
GetStorageStatsCommand .
Also see same exceptions for “GetHostStatsCommand”
2013-09-20 21:26:23,252 ERROR [vmware.resource.VmwareResource]
(DirectAgent-302:10.223.57.194) Unable to execute Ge
tStorageStatsCommand(storageId : 39e98257-9a36-3a72-afdc-f128c574c04f,
localPath: /export/home/sangeetha/campofinal
ga/primary2, poolType: NetworkFilesystem) due to Exception:
javax.xml.ws.soap.SOAPFaultException
Message:
Required property obj is missing from data object of type ObjectSpec
while parsing serialized DataObject of type
vmodl.query.PropertyCollector.ObjectSpec
at line 1, column 305
while parsing property "objectSet" of static type ArrayOfObjectSpec
while parsing serialized DataObject of type
vmodl.query.PropertyCollector.FilterSpec
at line 1, column 205
while parsing call information for method RetrieveProperties
at line 1, column 110
while parsing SOAP body
at line 1, column 102
while parsing SOAP envelope
at line 1, column 38
while parsing HTTP request for method retrieveContents
on object of type vmodl.query.PropertyCollector
at line 1, column 0
javax.xml.ws.soap.SOAPFaultException:
Required property obj is missing from data object of type ObjectSpec
while parsing serialized DataObject of type
vmodl.query.PropertyCollector.ObjectSpec
at line 1, column 305
while parsing property "objectSet" of static type ArrayOfObjectSpec
while parsing serialized DataObject of type
vmodl.query.PropertyCollector.FilterSpec
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
