Sanjeev N created CLOUDSTACK-3859:
-------------------------------------
Summary: [Object_Store_Refactor] Failed to attach uploaded volume
to guest vm
Key: CLOUDSTACK-3859
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3859
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Storage Controller, VMware, Volumes
Affects Versions: 4.2.0
Environment: Latest build from ACS 4.2 build
Storage: NFS for both primary and secondary
Cluster: VMWare
Reporter: Sanjeev N
Priority: Critical
Fix For: 4.2.0
[Object_Store_Refactor] Failed to attach uploaded volume to guest vm
Steps to Reproduce:
================
1.Bringup CS with vmware cluster
2.Deploy guest vm with default cent of template with only root disk
3.upload volume using API:
http://10.147.59.126:8096/client/api?command=uploadVolume&format=OVA&name=cent53-upload&url=http://10.147.28.7/templates/vmware/CentOS5.3-x86_64.ova&zoneid=614d5a1b-cdee-41ad-ae36-a16d6a3a4aa8&account=admin&domainid=1
4.Attach the uploaded volume to guest vm
Result:
======
Upload volume failed and here is the log snippet:
2013-07-26 09:41:15,370 DEBUG [cloud.async.AsyncJobManagerImpl]
(ApiServer-6:null) submit async job-47 = [ 54aaf13a-3f49-4eef-b298-5398ba8fe663
], details: AsyncJobVO {id:47, userId: 1, accountId: 1, sessionKey: null,
instanceType: Volume, instanceId: 19, cmd:
org.apache.cloudstack.api.command.user.volume.AttachVolumeCmd, cmdOriginator:
null, cmdInfo:
{"id":"066a9c8f-989e-406a-a7b8-4e5989ab462b","virtualmachineid":"ca332253-ec27-4a76-b447-e2f82dfd1463","cmdEventType":"VOLUME.ATTACH","ctxUserId":"1","httpmethod":"GET","ctxAccountId":"1","ctxStartEventId":"179"},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
processStatus: 0, resultCode: 0, result: null, initMsid: 6615759585382,
completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2013-07-26 09:41:15,382 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-47:job-47 = [ 54aaf13a-3f49-4eef-b298-5398ba8fe663 ]) Executing
org.apache.cloudstack.api.command.user.volume.AttachVolumeCmd for job-47 = [
54aaf13a-3f49-4eef-b298-5398ba8fe663 ]
2013-07-26 09:41:15,446 DEBUG [storage.allocator.LocalStoragePoolAllocator]
(Job-Executor-47:job-47 = [ 54aaf13a-3f49-4eef-b298-5398ba8fe663 ])
LocalStoragePoolAllocator trying to find storage pool to fit the vm
2013-07-26 09:41:15,447 DEBUG
[storage.allocator.ClusterScopeStoragePoolAllocator] (Job-Executor-47:job-47 =
[ 54aaf13a-3f49-4eef-b298-5398ba8fe663 ]) ClusterScopeStoragePoolAllocator
looking for storage pool
2013-07-26 09:41:15,448 DEBUG
[storage.allocator.ClusterScopeStoragePoolAllocator] (Job-Executor-47:job-47 =
[ 54aaf13a-3f49-4eef-b298-5398ba8fe663 ]) Looking for pools in dc: 2 pod:2
cluster:2
2013-07-26 09:41:15,454 DEBUG [storage.allocator.AbstractStoragePoolAllocator]
(Job-Executor-47:job-47 = [ 54aaf13a-3f49-4eef-b298-5398ba8fe663 ]) Checking if
storage pool is suitable, name: null ,poolId: 2
2013-07-26 09:41:15,460 DEBUG [cloud.storage.StorageManagerImpl]
(Job-Executor-47:job-47 = [ 54aaf13a-3f49-4eef-b298-5398ba8fe663 ]) Checking
pool 2 for storage, totalSize: 5902284800000, usedBytes: 3909444313088,
usedPct: 0.6623611780116067, disable threshold: 0.85
2013-07-26 09:41:15,493 DEBUG [cloud.storage.StorageManagerImpl]
(Job-Executor-47:job-47 = [ 54aaf13a-3f49-4eef-b298-5398ba8fe663 ]) Checking
pool: 2 for volume allocation [Vol[19|vm=null|DATADISK]], maxSize :
11804569600000, totalAllocatedSize : 16740, askingSize : 2147483648, allocated
disable threshold: 0.85
2013-07-26 09:41:15,493 DEBUG
[storage.allocator.ClusterScopeStoragePoolAllocator] (Job-Executor-47:job-47 =
[ 54aaf13a-3f49-4eef-b298-5398ba8fe663 ]) FirstFitStoragePoolAllocator
returning 1 suitable storage pools
2013-07-26 09:41:15,525 DEBUG [storage.motion.AncientDataMotionStrategy]
(Job-Executor-47:job-47 = [ 54aaf13a-3f49-4eef-b298-5398ba8fe663 ]) copyAsync
inspecting src type VOLUME copyAsync inspecting dest type VOLUME
2013-07-26 09:41:15,542 DEBUG [agent.transport.Request] (Job-Executor-47:job-47
= [ 54aaf13a-3f49-4eef-b298-5398ba8fe663 ]) Seq 2-1034028339: Sending { Cmd ,
MgmtId: 6615759585382, via: 2, Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"066a9c8f-989e-406a-a7b8-4e5989ab462b","volumeType":"DATADISK","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_esx_os","_role":"Image"}},"name":"cent53-upload","size":2147483648,"path":"volumes/2/19/07c251ba-6ef7-3a57-a4cb-b75cf8dfcc2c.ova","volumeId":19,"accountId":2,"format":"OVA","id":19,"hypervisorType":"None"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"066a9c8f-989e-406a-a7b8-4e5989ab462b","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"468ecd4b-14a6-3b7d-8991-c9dd7427b94a","id":2,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_esx_os","port":2049}},"name":"cent53-upload","size":2147483648,"volumeId":19,"accountId":2,"format":"OVA","id":19,"hypervisorType":"None"}},"executeInSequence":false,"wait":10800}}]
}
2013-07-26 09:41:15,543 DEBUG [agent.transport.Request] (Job-Executor-47:job-47
= [ 54aaf13a-3f49-4eef-b298-5398ba8fe663 ]) Seq 2-1034028339: Executing: { Cmd
, MgmtId: 6615759585382, via: 2, Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"066a9c8f-989e-406a-a7b8-4e5989ab462b","volumeType":"DATADISK","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_esx_os","_role":"Image"}},"name":"cent53-upload","size":2147483648,"path":"volumes/2/19/07c251ba-6ef7-3a57-a4cb-b75cf8dfcc2c.ova","volumeId":19,"accountId":2,"format":"OVA","id":19,"hypervisorType":"None"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"066a9c8f-989e-406a-a7b8-4e5989ab462b","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"468ecd4b-14a6-3b7d-8991-c9dd7427b94a","id":2,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_esx_os","port":2049}},"name":"cent53-upload","size":2147483648,"volumeId":19,"accountId":2,"format":"OVA","id":19,"hypervisorType":"None"}},"executeInSequence":false,"wait":10800}}]
}
2013-07-26 09:41:15,544 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-288:null) Seq 2-1034028339: Executing request
2013-07-26 09:41:15,570 ERROR [storage.resource.VmwareStorageProcessor]
(DirectAgent-288:10.147.40.13) Unable to find snapshot OVA file at:
/var/cloudstack/mnt/VM/6615759585382.733bb9c1/volumes/2/19/07c251ba-6ef7-3a57-a4cb-b75cf8dfcc2c.ova/07c251ba-6ef7-3a57-a4cb-b75cf8dfcc2c.ova.ova
2013-07-26 09:41:15,576 ERROR [storage.resource.VmwareStorageProcessor]
(DirectAgent-288:10.147.40.13) Unable to execute CopyVolumeCommand due to
exception
java.lang.Exception: Unable to find snapshot OVA file at:
/var/cloudstack/mnt/VM/6615759585382.733bb9c1/volumes/2/19/07c251ba-6ef7-3a57-a4cb-b75cf8dfcc2c.ova/07c251ba-6ef7-3a57-a4cb-b75cf8dfcc2c.ova.ova
at
com.cloud.storage.resource.VmwareStorageProcessor.restoreVolumeFromSecStorage(VmwareStorageProcessor.java:1573)
at
com.cloud.storage.resource.VmwareStorageProcessor.copyVolumeFromSecStorage(VmwareStorageProcessor.java:431)
at
com.cloud.storage.resource.VmwareStorageProcessor.copyVolumeFromImageCacheToPrimary(VmwareStorageProcessor.java:477)
at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:76)
at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
at
com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:566)
at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-07-26 09:41:15,578 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-288:null) Seq 2-1034028339: Response Received:
2013-07-26 09:41:15,578 DEBUG [agent.transport.Request] (DirectAgent-288:null)
Seq 2-1034028339: Processing: { Ans: , MgmtId: 6615759585382, via: 2, Ver: v1,
Flags: 10,
[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"CopyVolumeCommand
failed due to exception: java.lang.Exception: Unable to find snapshot OVA file
at:
/var/cloudstack/mnt/VM/6615759585382.733bb9c1/volumes/2/19/07c251ba-6ef7-3a57-a4cb-b75cf8dfcc2c.ova/07c251ba-6ef7-3a57-a4cb-b75cf8dfcc2c.ova.ova","wait":0}}]
}
2013-07-26 09:41:15,579 DEBUG [agent.transport.Request] (Job-Executor-47:job-47
= [ 54aaf13a-3f49-4eef-b298-5398ba8fe663 ]) Seq 2-1034028339: Received: { Ans:
, MgmtId: 6615759585382, via: 2, Ver: v1, Flags: 10, { CopyCmdAnswer } }
2013-07-26 09:41:15,588 DEBUG [cloud.storage.VolumeManagerImpl]
(Job-Executor-47:job-47 = [ 54aaf13a-3f49-4eef-b298-5398ba8fe663 ]) copy volume
failed: java.lang.NullPointerException
2013-07-26 09:41:15,597 ERROR [cloud.async.AsyncJobManagerImpl]
(Job-Executor-47:job-47 = [ 54aaf13a-3f49-4eef-b298-5398ba8fe663 ]) Unexpected
exception while executing
org.apache.cloudstack.api.command.user.volume.AttachVolumeCmd
com.cloud.utils.exception.CloudRuntimeException: copy volume failed:
java.lang.NullPointerException
at
com.cloud.storage.VolumeManagerImpl.copyVolumeFromSecToPrimary(VolumeManagerImpl.java:652)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
com.cloud.storage.VolumeManagerImpl.copyVolume(VolumeManagerImpl.java:1529)
at
com.cloud.storage.VolumeManagerImpl.createVolumeOnPrimaryStorage(VolumeManagerImpl.java:1560)
at
com.cloud.storage.VolumeManagerImpl.attachVolumeToVM(VolumeManagerImpl.java:1853)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.api.command.user.volume.AttachVolumeCmd.execute(AttachVolumeCmd.java:122)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-07-26 09:41:15,601 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-47:job-47 = [ 54aaf13a-3f49-4eef-b298-5398ba8fe663 ]) Complete
async job-47 = [ 54aaf13a-3f49-4eef-b298-5398ba8fe663 ], jobStatus: 2,
resultCode: 530, result: Error Code: 530 Error text: copy volume failed:
java.lang.NullPointerException
>From the log I could see that CopyVolumeCommand looking for .ova file at
>following location:
/var/cloudstack/mnt/VM/6615759585382.733bb9c1/volumes/2/19/07c251ba-6ef7-3a57-a4cb-b75cf8dfcc2c.ova/07c251ba-6ef7-3a57-a4cb-b75cf8dfcc2c.ova.ova
But .ova is present at
/var/cloudstack/mnt/VM/6615759585382.733bb9c1/volumes/2/19
Here is the files list from the secondary storage:
[root@Rhel63-Sanjeev 19]# pwd
/var/cloudstack/mnt/VM/6615759585382.733bb9c1/volumes/2/19
[root@Rhel63-Sanjeev 19]# ls -l
total 898008
-rw-rw-rw-+ 1 root root 459320832 Jul 25 21:31
07c251ba-6ef7-3a57-a4cb-b75cf8dfcc2c.ova
-rw-r-----+ 1 root root 459312128 Sep 17 2010 CentOS5.3-x86_64-disk1.vmdk
-rw-r-----+ 1 root root 147 Sep 17 2010 CentOS5.3-x86_64.mf
-rw-r-----+ 1 root root 5340 Sep 17 2010 CentOS5.3-x86_64.ovf
-rw-rw-rw-+ 1 root root 337 Jul 25 21:32 volume.properties
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
