Sanjeev N created CLOUDSTACK-3737:
-------------------------------------
Summary: Uploaded volume is not getting deleted from secondary
storage after attaching it to guest vm
Key: CLOUDSTACK-3737
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3737
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Storage Controller, Volumes
Affects Versions: 4.2.0
Environment: Latest build from ACS 4.2 branch
Storage: NFS for both primary and secondary
Reporter: Sanjeev N
Fix For: 4.2.0
Uploaded volume is not getting deleted from secondary storage after attaching
it to guest vm
Steps to Reproduce:
================
1.Bring up CS with xen hypervisor and use NFS as the storage for both primary
and secondary storage
2.Deploy guest vm using default cent of template just with root disk
3.Upload volume to CS using following API:
http://10.147.59.126:8096/client/api?command=uploadVolume&format=VHD&name=cent62&url=http://10.147.28.7/templates/CentOS62-64bit/280c2a70-e37f-4863-bff8-d318122fd61b.vhd&zoneid=5c5c0b8a-9d5a-4b95-8f13-b31058ffdb37&account=admin&domainid=1
4.After the volume download is complete to secondary storage , attache it to vm
using API:
http://10.147.59.126:8096/client/api?command=attachVolume&id=0d627eec-3824-4d35-8997-853472502454&virtualmachineid=6ef83f81-2577-4be2-9720-5fdcf3912e5f
Observations:
===========
After step4 volume should be moved from secondary to primary. But the
observation is it is just getting copied to primary and still present in the
secondary. It should be deleted from secondary storage after successful copying
to primary storage.
Volume state in DB after attaching to VM:
mysql> select * from volumes where
uuid='0d627eec-3824-4d35-8997-853472502454'\G;
*************************** 1. row ***************************
id: 10
account_id: 2
domain_id: 1
pool_id: 1
last_pool_id: NULL
instance_id: 3
device_id: 1
name: cent62
uuid: 0d627eec-3824-4d35-8997-853472502454
size: 10737418240
folder: NULL
path: b2b96197-665a-40cd-a52b-de0506c45a8e
pod_id: NULL
data_center_id: 1
iscsi_name: NULL
host_ip: NULL
volume_type: DATADISK
pool_type: NULL
disk_offering_id: 6
template_id: NULL
first_snapshot_backup_uuid: NULL
recreatable: 0
created: 2013-07-23 12:36:22
attached: 2013-07-23 12:52:40
updated: 2013-07-23 13:09:08
removed: NULL
state: Ready
chain_info: NULL
update_count: 6
disk_type: NULL
display_volume: 0
format: VHD
min_iops: NULL
max_iops: NULL
1 row in set (0.00 sec)
ERROR:
No query specified
In volume_store_ref the volume state is remained in "Creating" state. Still
attaching volume was succeeded.
Here is the volume state in volume_store_ref from cloud db:
mysql> select * from volume_store_ref where id=4\G;
*************************** 1. row ***************************
id: 4
store_id: 1
volume_id: 10
zone_id: 0
created: 2013-07-23 13:09:07
last_updated: NULL
job_id: NULL
download_pct: 0
size: 0
physical_size: 0
download_state: NULL
checksum: NULL
error_str: NULL
local_path: NULL
install_path: volumes/2/10
url: NULL
download_url: NULL
state: Creating
destroyed: 0
update_count: 1
ref_cnt: 0
updated: 2013-07-23 13:09:07
1 row in set (0.00 sec)
ERROR:
No query specified
Log snippet for upload volume:
2013-07-23 08:36:22,827 DEBUG [cloud.async.AsyncJobManagerImpl]
(ApiServer-1:null) submit async job-22 = [ 63bf80a6-5142-489e-ab5d-7a5ef061140d
], details: AsyncJobVO {id:22, userId: 1, accountId: 1, sessionKey: null,
instanceType: None, instanceId: null, cmd:
org.apache.cloudstack.api.command.user.volume.UploadVolumeCmd, cmdOriginator:
null, cmdInfo:
{"cmdEventType":"VOLUME.UPLOAD","ctxUserId":"1","zoneid":"5c5c0b8a-9d5a-4b95-8f13-b31058ffdb37","name":"cent62","account":"admin","httpmethod":"GET","domainid":"1","ctxAccountId":"1","format":"VHD","ctxStartEventId":"91","url":"http://10.147.28.7/templates/CentOS62-64bit/280c2a70-e37f-4863-bff8-d318122fd61b.vhd"},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
processStatus: 0, resultCode: 0, result: null, initMsid: 6615759585382,
completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2013-07-23 08:36:22,837 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-22:job-22 = [ 63bf80a6-5142-489e-ab5d-7a5ef061140d ]) Executing
org.apache.cloudstack.api.command.user.volume.UploadVolumeCmd for job-22 = [
63bf80a6-5142-489e-ab5d-7a5ef061140d ]
2013-07-23 08:36:23,066 DEBUG [storage.image.BaseImageStoreDriverImpl]
(Job-Executor-22:job-22 = [ 63bf80a6-5142-489e-ab5d-7a5ef061140d ]) Downloading
volume to data store 1
2013-07-23 08:36:23,121 DEBUG [cloudstack.storage.RemoteHostEndPoint]
(Job-Executor-22:job-22 = [ 63bf80a6-5142-489e-ab5d-7a5ef061140d ]) Sending
command org.apache.cloudstack.storage.command.DownloadCommand to host: 3
2013-07-23 08:36:23,125 DEBUG [agent.transport.Request] (Job-Executor-22:job-22
= [ 63bf80a6-5142-489e-ab5d-7a5ef061140d ]) Seq 3-1340014758: Sending { Cmd ,
MgmtId: 6615759585382, via: 3, Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.DownloadCommand":{"hvm":false,"maxDownloadSizeInBytes":536870912000,"id":10,"resourceType":"VOLUME","installPath":"volumes/2/10","_store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_xen_os","_role":"Image"}},"url":"http://10.147.28.7/templates/CentOS62-64bit/280c2a70-e37f-4863-bff8-d318122fd61b.vhd","format":"VHD","accountId":2,"name":"cent62","wait":0}}]
}
2013-07-23 08:36:23,149 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-22:job-22 = [ 63bf80a6-5142-489e-ab5d-7a5ef061140d ]) Complete
async job-22 = [ 63bf80a6-5142-489e-ab5d-7a5ef061140d ], jobStatus: 1,
resultCode: 0, result:
org.apache.cloudstack.api.response.VolumeResponse@2a5b5372
2013-07-23 08:36:23,163 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-22:job-22 = [ 63bf80a6-5142-489e-ab5d-7a5ef061140d ]) Done
executing org.apache.cloudstack.api.command.user.volume.UploadVolumeCmd for
job-22 = [ 63bf80a6-5142-489e-ab5d-7a5ef061140d ]
2013-07-23 08:36:26,660 DEBUG [agent.transport.Request]
(AgentManager-Handler-14:null) Seq 3-1340014758: Processing: { Ans: , MgmtId:
6615759585382, via: 3, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.storage.DownloadAnswer":{"jobId":"9d1225f8-4dd3-44fb-b39d-391583e95281","downloadPct":1,"errorString":"
","downloadStatus":"DOWNLOAD_IN_PROGRESS","downloadPath":"/mnt/SecStorage/1f24310b-8f0b-3563-aecc-779179564245/volumes/2/10/dnld6777686660366560545tmp_","installPath":"volumes/2/10","templateSize":0,"templatePhySicalSize":0,"result":true,"details":"
","wait":0}}] }
After some time volume was successfully downloaded to secondary storage:
2013-07-23 08:45:31,785 DEBUG [agent.transport.Request]
(AgentManager-Handler-4:null) Seq 3-1340014793: Processing: { Ans: , MgmtId:
6615759585382, via: 3, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.storage.DownloadAnswer":{"jobId":"9d1225f8-4dd3-44fb-b39d-391583e95281","downloadPct":100,"errorString":"Install
completed successfully at 7/23/13 12:45
PM","downloadStatus":"DOWNLOADED","downloadPath":"/mnt/SecStorage/1f24310b-8f0b-3563-aecc-779179564245/volumes/2/10/dnld6777686660366560545tmp_","installPath":"volumes/2/10/a58b5e0e-8548-34e3-a22f-8a4b52701ac4.vhd","templateSize":10737418240,"templatePhySicalSize":2132787712,"checkSum":"bebf14997dd834de2053ce2dd6a903b6","result":true,"details":"Install
completed successfully at 7/23/13 12:45 PM","wait":0}}] }
Executed Attachevolume cmd to attach it to guest vm:
2013-07-23 08:50:36,484 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-23:job-23 = [ 45a8259a-4c9f-4725-bb73-43fb4d0e786b ]) Executing
org.apache.cloudstack.api.command.user.volume.AttachVolumeCmd for job-23 = [
45a8259a-4c9f-4725-bb73-43fb4d0e786b ]
2013-07-23 08:50:36,549 DEBUG [storage.allocator.LocalStoragePoolAllocator]
(Job-Executor-23:job-23 = [ 45a8259a-4c9f-4725-bb73-43fb4d0e786b ])
LocalStoragePoolAllocator trying to find storage pool to fit the vm
2013-07-23 08:50:36,551 DEBUG
[storage.allocator.ClusterScopeStoragePoolAllocator] (Job-Executor-23:job-23 =
[ 45a8259a-4c9f-4725-bb73-43fb4d0e786b ]) ClusterScopeStoragePoolAllocator
looking for storage pool
2013-07-23 08:50:36,551 DEBUG
[storage.allocator.ClusterScopeStoragePoolAllocator] (Job-Executor-23:job-23 =
[ 45a8259a-4c9f-4725-bb73-43fb4d0e786b ]) Looking for pools in dc: 1 pod:1
cluster:1
2013-07-23 08:50:36,559 DEBUG [storage.allocator.AbstractStoragePoolAllocator]
(Job-Executor-23:job-23 = [ 45a8259a-4c9f-4725-bb73-43fb4d0e786b ]) Checking if
storage pool is suitable, name: null ,poolId: 1
2013-07-23 08:50:36,567 DEBUG [cloud.storage.StorageManagerImpl]
(Job-Executor-23:job-23 = [ 45a8259a-4c9f-4725-bb73-43fb4d0e786b ]) Checking
pool 1 for storage, totalSize: 5902284816384, usedBytes: 3927302963200,
usedPct: 0.6653868942919022, disable threshold: 0.85
2013-07-23 08:50:36,598 DEBUG [cloud.storage.StorageManagerImpl]
(Job-Executor-23:job-23 = [ 45a8259a-4c9f-4725-bb73-43fb4d0e786b ]) Checking
pool: 1 for volume allocation [Vol[10|vm=null|DATADISK]], maxSize :
11804569632768, totalAllocatedSize : 54609838080, askingSize : 10737418240,
allocated disable threshold: 0.85
2013-07-23 08:50:36,598 DEBUG
[storage.allocator.ClusterScopeStoragePoolAllocator] (Job-Executor-23:job-23 =
[ 45a8259a-4c9f-4725-bb73-43fb4d0e786b ]) FirstFitStoragePoolAllocator
returning 1 suitable storage pools
2013-07-23 08:50:36,638 DEBUG [storage.motion.AncientDataMotionStrategy]
(Job-Executor-23:job-23 = [ 45a8259a-4c9f-4725-bb73-43fb4d0e786b ]) copyAsync
inspecting src type VOLUME copyAsync inspecting dest type VOLUME
2013-07-23 08:50:36,653 DEBUG [agent.transport.Request] (Job-Executor-23:job-23
= [ 45a8259a-4c9f-4725-bb73-43fb4d0e786b ]) Seq 1-1929052680: Sending { Cmd ,
MgmtId: 6615759585382, via: 1, Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"0d627eec-3824-4d35-8997-853472502454","volumeType":"DATADISK","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_xen_os","_role":"Image"}},"name":"cent62","size":10737418240,"path":"volumes/2/10/a58b5e0e-8548-34e3-a22f-8a4b52701ac4.vhd","volumeId":10,"accountId":2,"format":"VHD","id":10}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"0d627eec-3824-4d35-8997-853472502454","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c65a038a-750c-3b4f-bf26-7ce3b74e1c85","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_xen_os","port":2049}},"name":"cent62","size":10737418240,"volumeId":10,"accountId":2,"format":"VHD","id":10}},"executeInSequence":false,"wait":10800}}]
}
2013-07-23 08:50:36,654 DEBUG [agent.transport.Request] (Job-Executor-23:job-23
= [ 45a8259a-4c9f-4725-bb73-43fb4d0e786b ]) Seq 1-1929052680: Executing: { Cmd
, MgmtId: 6615759585382, via: 1, Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"0d627eec-3824-4d35-8997-853472502454","volumeType":"DATADISK","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_xen_os","_role":"Image"}},"name":"cent62","size":10737418240,"path":"volumes/2/10/a58b5e0e-8548-34e3-a22f-8a4b52701ac4.vhd","volumeId":10,"accountId":2,"format":"VHD","id":10}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"0d627eec-3824-4d35-8997-853472502454","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c65a038a-750c-3b4f-bf26-7ce3b74e1c85","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_xen_os","port":2049}},"name":"cent62","size":10737418240,"volumeId":10,"accountId":2,"format":"VHD","id":10}},"executeInSequence":false,"wait":10800}}]
}
2013-07-23 08:50:36,655 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-17:null) Seq 1-1929052680: Executing request
2013-07-23 08:52:39,712 DEBUG [agent.transport.Request] (Job-Executor-23:job-23
= [ 45a8259a-4c9f-4725-bb73-43fb4d0e786b ]) Seq 1-1929052687: Sending { Cmd ,
MgmtId: 6615759585382, via: 1, Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.AttachCommand":{"disk":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"0d627eec-3824-4d35-8997-853472502454","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c65a038a-750c-3b4f-bf26-7ce3b74e1c85","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_xen_os","port":2049}},"name":"cent62","size":10737418240,"path":"b2b96197-665a-40cd-a52b-de0506c45a8e","volumeId":10,"accountId":2,"format":"VHD","id":10}},"diskSeq":1,"type":"DATADISK"},"vmName":"i-2-3-VM","_storageHost":"10.147.28.7","_storagePort":2049,"_managed":false,"wait":0}}]
}
2013-07-23 08:52:39,713 DEBUG [agent.transport.Request] (Job-Executor-23:job-23
= [ 45a8259a-4c9f-4725-bb73-43fb4d0e786b ]) Seq 1-1929052687: Executing: { Cmd
, MgmtId: 6615759585382, via: 1, Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.AttachCommand":{"disk":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"0d627eec-3824-4d35-8997-853472502454","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c65a038a-750c-3b4f-bf26-7ce3b74e1c85","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_xen_os","port":2049}},"name":"cent62","size":10737418240,"path":"b2b96197-665a-40cd-a52b-de0506c45a8e","volumeId":10,"accountId":2,"format":"VHD","id":10}},"diskSeq":1,"type":"DATADISK"},"vmName":"i-2-3-VM","_storageHost":"10.147.28.7","_storagePort":2049,"_managed":false,"wait":0}}]
}
2013-07-23 08:52:39,714 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-399:null) Seq 1-1929052687: Executing request
2013-07-23 08:52:40,868 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-399:null) Seq 1-1929052687: Response Received:
2013-07-23 08:52:40,870 DEBUG [agent.transport.Request] (DirectAgent-399:null)
Seq 1-1929052687: Processing: { Ans: , MgmtId: 6615759585382, via: 1, Ver: v1,
Flags: 10,
[{"org.apache.cloudstack.storage.command.AttachAnswer":{"disk":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"0d627eec-3824-4d35-8997-853472502454","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c65a038a-750c-3b4f-bf26-7ce3b74e1c85","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_xen_os","port":2049}},"name":"cent62","size":10737418240,"path":"b2b96197-665a-40cd-a52b-de0506c45a8e","volumeId":10,"accountId":2,"format":"VHD","id":10}},"diskSeq":1,"vdiUuid":"b2b96197-665a-40cd-a52b-de0506c45a8e","type":"DATADISK"},"result":true,"wait":0}}]
}
2013-07-23 08:52:40,870 DEBUG [agent.transport.Request] (Job-Executor-23:job-23
= [ 45a8259a-4c9f-4725-bb73-43fb4d0e786b ]) Seq 1-1929052687: Received: { Ans:
, MgmtId: 6615759585382, via: 1, Ver: v1, Flags: 10, { AttachAnswer } }
2013-07-23 08:52:40,915 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-23:job-23 = [ 45a8259a-4c9f-4725-bb73-43fb4d0e786b ]) Complete
async job-23 = [ 45a8259a-4c9f-4725-bb73-43fb4d0e786b ], jobStatus: 1,
resultCode: 0, result:
org.apache.cloudstack.api.response.VolumeResponse@50be7f47
2013-07-23 08:52:40,922 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-23:job-23 = [ 45a8259a-4c9f-4725-bb73-43fb4d0e786b ]) Done
executing org.apache.cloudstack.api.command.user.volume.AttachVolumeCmd for
job-23 = [ 45a8259a-4c9f-4725-bb73-43fb4d0e786b ]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
