[
https://issues.apache.org/jira/browse/CLOUDSTACK-6460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14143170#comment-14143170
]
Simon Weller commented on CLOUDSTACK-6460:
------------------------------------------
I added some debugging into getStoragePool, and createPhysicalDisk.
Agent logs in debug:
2014-09-22 07:03:50,164{GMT} DEBUG [cloud.agent.Agent]
(agentRequest-Handler-4:) Request:Seq 3-192151576: { Cmd , MgmtId:
144343444026, via: 3, Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"a8dbbcb0-5f27-470a-a482-e00ac0e68a75","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"076d1cd7-9c80-4302-8e13-ea8187a9a96b","id":201,"poolType":"CLVM","host":"localhost","path":"/csstore01","port":0,"url":"CLVM://localhost//csstore01/?ROLE=Primary&STOREUUID=076d1cd7-9c80-4302-8e13-ea8187a9a96b"}},"name":"test10_2","size":5368709120,"volumeId":17461,"accountId":6,"id":17461,"hypervisorType":"KVM"}},"wait":0}}]
}
2014-09-22 07:03:50,165{GMT} DEBUG [cloud.agent.Agent]
(agentRequest-Handler-4:) Processing command:
org.apache.cloudstack.storage.command.CreateObjectCommand
2014-09-22 07:03:50,172{GMT} DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:) getStoragePool: logical
2014-09-22 07:03:52,044{GMT} DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:) createPhysicalDisk format is raw
2014-09-22 07:03:52,045{GMT} DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:) <volume>
<name>a8dbbcb0-5f27-470a-a482-e00ac0e68a75</name>
<capacity>5368709120</capacity>
<target>
<format type='raw'/>
<permissions><mode>0744</mode></permissions></target>
</volume>
Mgmt logs:
2014-09-22 07:03:50,133 DEBUG [c.c.s.StorageManagerImpl]
(Job-Executor-7:ctx-e61f9a16 ctx-e997a276) Checking pool: 201 for volume
allocation [Vol[17461|vm=null|DATADISK]], maxSize : 1099507433472,
totalAllocatedSize : 83823165440, askingSize : 5368709120, allocated disable
threshold: 0.85
2014-09-22 07:03:50,133 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator]
(Job-Executor-7:ctx-e61f9a16 ctx-e997a276) ClusterScopeStoragePoolAllocator
returning 1 suitable storage pools
2014-09-22 07:03:50,133 DEBUG [o.a.c.e.o.VolumeOrchestrator]
(Job-Executor-7:ctx-e61f9a16 ctx-e997a276) Trying to create
org.apache.cloudstack.storage.volume.VolumeObject@76e38e5e on
org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@5597fa4c
2014-09-22 07:03:50,147 DEBUG
[o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (Job-Executor-7:ctx-e61f9a16
ctx-e997a276) Creating volume:
org.apache.cloudstack.storage.volume.VolumeObject@53789cfc
2014-09-22 07:03:50,151 DEBUG [c.c.a.t.Request] (Job-Executor-7:ctx-e61f9a16
ctx-e997a276) Seq 3-192151576: Sending { Cmd , MgmtId: 144343444026, via:
3(csh01-lab.zone01.nsvltn.cloud.ena.net), Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"a8dbbcb0-5f27-470a-a482-e00ac0e68a75","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"076d1cd7-9c80-4302-8e13-ea8187a9a96b","id":201,"poolType":"CLVM","host":"localhost","path":"/csstore01","port":0,"url":"CLVM://localhost//csstore01/?ROLE=Primary&STOREUUID=076d1cd7-9c80-4302-8e13-ea8187a9a96b"}},"name":"test10_2","size":5368709120,"volumeId":17461,"accountId":6,"id":17461,"hypervisorType":"KVM"}},"wait":0}}]
}
2014-09-22 07:03:52,871 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-7:null) SeqA 9-1551798: Processing Seq 9-1551798: { Cmd
, MgmtId: -1, via: 9, Ver: v1, Flags: 11,
[{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":22358,"_loadInfo":"{\n
\"connections\": []\n}","wait":0}}] }
2014-09-22 07:03:52,873 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-7:null) SeqA 9-1551798: Sending Seq 9-1551798: { Ans: ,
MgmtId: 144343444026, via: 9, Ver: v1, Flags: 100010,
[{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
2014-09-22 07:03:53,145 DEBUG [c.c.a.ApiServlet] (catalina-exec-4:ctx-27d5d351)
===START=== 172.27.2.190 -- GET
command=queryAsyncJobResult&jobId=3b9a0e48-cca9-49bd-8ba5-f3ee34fecd0a&response=json&sessionkey=7fKPZgeg1u%2Bw7dPiPjqnCTze56k%3D&projectid=729be1b4-429f-410c-b4fb-a9f5bfa3ec50&_=1411387433075
2014-09-22 07:03:53,154 DEBUG [c.c.a.ApiServlet] (catalina-exec-4:ctx-27d5d351
ctx-01bf906b) ===END=== 172.27.2.190 -- GET
command=queryAsyncJobResult&jobId=3b9a0e48-cca9-49bd-8ba5-f3ee34fecd0a&response=json&sessionkey=7fKPZgeg1u%2Bw7dPiPjqnCTze56k%3D&projectid=729be1b4-429f-410c-b4fb-a9f5bfa3ec50&_=1411387433075
2014-09-22 07:03:53,613 DEBUG [c.c.a.t.Request] (AgentManager-Handler-1:null)
Seq 3-192151576: Processing: { Ans: , MgmtId: 144343444026, via: 3, Ver: v1,
Flags: 10,
[{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"size":5368709120,"path":"a8dbbcb0-5f27-470a-a482-e00ac0e68a75","accountId":0,"format":"RAW","id":0}},"result":true,"wait":0}}]
}
2014-09-22 07:03:53,613 DEBUG [c.c.a.t.Request] (Job-Executor-7:ctx-e61f9a16
ctx-e997a276) Seq 3-192151576: Received: { Ans: , MgmtId: 144343444026, via:
3, Ver: v1, Flags: 10, { CreateObjectAnswer } }
2014-09-22 07:03:53,623 INFO [c.c.s.VolumeApiServiceImpl]
(Job-Executor-7:ctx-e61f9a16 ctx-e997a276) Trying to attaching volume 17461 to
vm instance:22370, update async job-922 progress status
2014-09-22 07:03:53,623 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Job-Executor-7:ctx-e61f9a16 ctx-e997a276) Update async-job attachment,
job-922, instanceType: volume, instanceId: 17461
2014-09-22 07:03:53,632 DEBUG [c.c.a.t.Request] (Job-Executor-7:ctx-e61f9a16
ctx-e997a276) Seq 1-1638137881: Sending { Cmd , MgmtId: 144343444026, via:
1(csh02-lab.zone01.nsvltn.cloud.ena.net), Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.AttachCommand":{"disk":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"a8dbbcb0-5f27-470a-a482-e00ac0e68a75","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"076d1cd7-9c80-4302-8e13-ea8187a9a96b","id":201,"poolType":"CLVM","host":"localhost","path":"/csstore01","port":0,"url":"CLVM://localhost//csstore01/?ROLE=Primary&STOREUUID=076d1cd7-9c80-4302-8e13-ea8187a9a96b"}},"name":"test10_2","size":5368709120,"path":"a8dbbcb0-5f27-470a-a482-e00ac0e68a75","volumeId":17461,"accountId":6,"format":"QCOW2","id":17461,"hypervisorType":"KVM"}},"diskSeq":6,"path":"a8dbbcb0-5f27-470a-a482-e00ac0e68a75","type":"DATADISK","_details":{"managed":"false","storagePort":"0","storageHost":"localhost","volumeSize":"5368709120"}},"vmName":"i-6-22370-VM","inSeq":false,"wait":0}}]
}
2014-09-22 07:03:55,557 DEBUG [c.c.a.t.Request] (AgentManager-Handler-8:null)
Seq 1-1638137881: Processing: { Ans: , MgmtId: 144343444026, via: 1, Ver: v1,
Flags: 10,
[{"org.apache.cloudstack.storage.command.AttachAnswer":{"disk":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"a8dbbcb0-5f27-470a-a482-e00ac0e68a75","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"076d1cd7-9c80-4302-8e13-ea8187a9a96b","id":201,"poolType":"CLVM","host":"localhost","path":"/csstore01","port":0,"url":"CLVM://localhost//csstore01/?ROLE=Primary&STOREUUID=076d1cd7-9c80-4302-8e13-ea8187a9a96b"}},"name":"test10_2","size":5368709120,"path":"a8dbbcb0-5f27-470a-a482-e00ac0e68a75","volumeId":17461,"accountId":6,"format":"QCOW2","id":17461,"hypervisorType":"KVM"}},"diskSeq":6,"path":"a8dbbcb0-5f27-470a-a482-e00ac0e68a75","type":"DATADISK","_details":{"managed":"false","storagePort":"0","storageHost":"localhost","volumeSize":"5368709120"}},"result":true,"wait":0}}]
}
2014-09-22 07:03:55,557 DEBUG [c.c.a.t.Request] (Job-Executor-7:ctx-e61f9a16
ctx-e997a276) Seq 1-1638137881: Received: { Ans: , MgmtId: 144343444026, via:
1, Ver: v1, Flags: 10, { AttachAnswer } }
2014-09-22 07:03:55,564 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Job-Executor-7:ctx-e61f9a16 ctx-e997a276) Complete async job-922, jobStatus:
SUCCEEDED, resultCode: 0, result:
org.apache.cloudstack.api.response.VolumeResponse/volume/{"id":"a8dbbcb0-5f27-470a-a482-e00ac0e68a75","name":"test10_2","zoneid":"d6b4c47c-9df3-4c74-aacb-7960111acb25","zonename":"Zone1","type":"DATADISK","deviceid":6,"virtualmachineid":"1d752c7b-87a7-46dd-a987-65c79f914244","vmname":"linux-test-20G","vmdisplayname":"linux-test-20G","vmstate":"Running","size":5368709120,"created":"2014-09-22T07:03:14-0500","state":"Ready","projectid":"729be1b4-429f-410c-b4fb-a9f5bfa3ec50","project":"test","domainid":"8ece7bda-7e2a-11e3-8362-00219b8a2a3a","domain":"ROOT","storagetype":"shared","hypervisor":"KVM","diskofferingid":"1155bee8-350c-4e71-9958-83c3f44935ea","diskofferingname":"Small","diskofferingdisplaytext":"Small
Disk, 5
GB","storage":"clvm1","attached":"2014-09-22T07:03:55-0500","destroyed":false,"isextractable":true,"tags":[],"displayvolume":true,"path":"a8dbbcb0-5f27-470a-a482-e00ac0e68a75","storageid":"076d1cd7-9c80-4302-8e13-ea8187a9a96b","quiescevm":false,"jobid":"3b9a0e48-cca9-49bd-8ba5-f3ee34fecd0a","jobstatus":0}
2014-09-22 07:03:55,567 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Job-Executor-7:ctx-e61f9a16) Done executing
org.apache.cloudstack.api.command.user.volume.AttachVolumeCmd for job-922
So it appears as if the agent code is working as expected. getStoragePool is
seeing the CLVM storage as logical, and when I try and create the physical
disk, it's being set as format raw. Yet the mgmt server is returning QCOW2 as
the format.
> Migration of CLVM volumes to another primary storage fail
> ---------------------------------------------------------
>
> Key: CLOUDSTACK-6460
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-6460
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: KVM, Volumes
> Affects Versions: 4.2.0, 4.2.1, 4.3.0, 4.4.0
> Environment: KVM clusters with fiber channel SAN storage, CLVM volumes
> Reporter: Salvatore Sciacco
> Attachments: CLOUDSTACK-6460-darrentang.patch,
> CLOUDSTACK-6460-darrentang_2.patch, cloudstack-6460.patch,
> cloudstack-6460_44.patch
>
>
> ACS version: 4.2.1
> Hypervisors: KVM
> Storage pool type: CLVM
> Since we upgraded from 4.1 to 4.2.1 moving volumes to a different primary
> storage pool fail. I've enabled debug on the agents side and I think there is
> a problem with the format type conversion
> Volume on database had format QCOW2
> these are the parameters for the first step (CLVM -> NFS):
> {quote}
> "srcTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":
> "uuid":"cda46430-52d7-4bf0-b0c2-adfc78dd011c","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":"uuid":"655d6965-b3f3-4118-a970-d50cf6afc365","id":211,"poolType":"CLVM","host":"localhost","path":"/FC10KY1","port":0,"name":"ROOT-4450","size":5368709120,"path":"39a25daf-23a1-4b65-99ac-fb98469ac197","volumeId":5937,"vmName":"i-402-4450-VM","accountId":402,"format":"QCOW2","id":5937,"hypervisorType":"KVM"}
> "destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"cda46430-52d7-4bf0-b0c2-adfc78dd011c","volumeType":"ROOT","dataStore":{"com.cloud.agent.api.to.NfsTO":
>
> "_url":"nfs://192.168.11.6/home/a1iwstack","_role":"Image"},"name":"ROOT-4450","size":5368709120,"path":"volumes/402/5937","volumeId":5937,"vmName":"i-402-4450-VM","accountId":402,"format":"QCOW2","id":5937,"hypervisorType":"KVM"}
> {quote}
> Those commads are translated into the agent:
> {quote}
> DEBUG [utils.script.Script] (agentRequest-Handler-1:null) Executing: qemu-img
> info /dev/FC10KY1/39a25daf-23a1-4b65-99ac-fb98469ac197
> DEBUG [utils.script.Script] (agentRequest-Handler-1:null) Execution is
> successful.
> DEBUG [utils.script.Script] (agentRequest-Handler-1:null) Executing:
> /bin/bash -c cp -f /dev/FC10KY1/39a25daf-23a1-4b65-99ac-fb98469ac197
> /mnt/b8311c72-fe75-3832-98fc-975445028a12/5c713376-c418-478c-8a31-89c4181cb48e.qcow2
>
> {quote}
> With the result that the output file isn't a qcow2 file but a raw partition,
> which in turn make the next step fail.
> (NFS -> CLVM)
> {quote}
> DEBUG [utils.script.Script] (agentRequest-Handler-2:) Executing: qemu-img
> info
> /mnt/b8311c72-fe75-3832-98fc-975445028a12/b9303d8d-cd51-4b6c-a244-43c405df4238.qcow2
>
> DEBUG [utils.script.Script] (agentRequest-Handler-2:) Execution is successful.
> DEBUG [utils.script.Script] (agentRequest-Handler-2:) Executing: qemu-img
> convert -f qcow2 -O raw
> /mnt/b8311c72-fe75-3832-98fc-975445028a12/b9303d8d-cd51-4b6c-a244-43c405df4238.qcow2
> /dev/FCSTORAGE/da162325-467b-4e78-af07-4bad85470d66
> DEBUG [utils.script.Script] (agentRequest-Handler-2:) Exit value is 1
> DEBUG [utils.script.Script] (agentRequest-Handler-2:) qemu-img: Could not
> open
> '/mnt/b8311c72-fe75-3832-98fc-975445028a12/b9303d8d-cd51-4b6c-a244-43c405df4238.qcow2'qemu-img:
> Could not open
> '/mnt/b8311c72-fe75-3832-98fc-975445028a12/b9303d8d-cd51-4b6c-a244-43c405df4238.qcow2'
> ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:) Failed to
> convert
> /mnt/b8311c72-fe75-3832-98fc-975445028a12/b9303d8d-cd51-4b6c-a244-43c405df4238.qcow2
> to /dev/FCSTORAGE/da162325-467b-4e78-af07-4bad85470d66 the error was:
> qemu-img: Could not open
> '/mnt/b8311c72-fe75-3832-98fc-975445028a12/b9303d8d-cd51-4b6c-a244-43c405df4238.qcow2'qemu-img:
> Could not open
> '/mnt/b8311c72-fe75-3832-98fc-975445028a12/b9303d8d-cd51-4b6c-a244-43c405df4238.qcow2'
> {quote}
> If I change on the database the format of the volume to RAW the effect is
> even worse as data is lost in the process!
> These are the parameter for the first step (CLVM => NFS)
> {quote}
> "srcTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"cda46430-52d7-4bf0-b0c2-adfc78dd011c","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"655d6965-b3f3-4118-a970d50cf6afc365","id":211,"poolType":"CLVM","host":"localhost","path":"/FC10KY1","port":0,"name":"ROOT-4450"
> ,"size":5368709120,"path":"39a25daf-23a1-4b65-99ac-fb98469ac197","volumeId":5937,"vmName":"i-4024450VM","accountId":402,"format":"RAW","id":5937,"hypervisorType":"KVM"
> "destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":"uuid":"cda46430-52d7-4bf0-b0c2-adfc78dd011c","volumeType":"ROOT","dataStore":{"com.cloud.agent.api.to.NfsTO":
>
> "_url":"nfs://192.168.11.6/home/a1iwstack","_role":"Image"}},"name":"ROOT4450","size":5368709120,"path":"volumes/402/5937","volumeId":5937,"vmName":"i-402-4450-VM","accountId":402,"format":"RAW","id":5937,"hypervisorType":"KVM"}
> {quote}
> this time the output is converted to qcow2!
> {quote}
> DEBUG [utils.script.Script] (agentRequest-Handler-3:null) Executing: qemu-img
> info /dev/FC10KY1/39a25daf-23a1-4b65-99ac-fb98469ac197
> DEBUG [utils.script.Script] (agentRequest-Handler-3:null) Execution is
> successful.
> DEBUG [utils.script.Script] (agentRequest-Handler-3:null) Executing: qemu-img
> convert -f raw -O qcow2 /dev/FC10KY1/39a25daf-23a1-4b65-99ac-fb98469ac197
> /mnt/b8311c72-fe75-3832-98fc-975445028a12/01ab129f-aaf6-4b1a-8e2a-093bee0b811c.raw
>
> {quote}
> and data is lost in the next step (NFS -> CLVM):
> {quote}
> "srcTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"cda46430-52d7-4bf0-b0c2-adfc78dd011c","volumeType":"ROOT","dataStore":{"com.cl
> oud.agent.api.to.NfsTO":{"_url":"nfs://192.168.11.6/home/a1iwstack","_role":"Image"}},"name":"ROOT4450","size":5368709120,"path":"volumes/402/5937/01ab129f-aaf6-4b1a-8e2a-093bee0b811c.raw","volumeId":5937,"vmName":"i-402-4450-VM","accountId":402,"format":"RAW","id":5937,"hypervisorType":"KVM"
> "destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"2d684689-efbc-4ce3-95ad-b38e80a1afcf","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4d59fde3-4c93-4aa5-8041-fab797b90899","id":214,"poolType":
> "CLVM","host":"localhost","path":"/STOREY1","port":0}},"name":"ROOT-4450","size":5368709120,"path":"39a25daf-23a1-4b65-99ac-fb98469ac197","volumeId":5967,"vmName":"i-402-4450-VM","accountId":402,"format":"RAW","id":5967,"hypervisorType":"KVM"
> DEBUG [utils.script.Script] (agentRequest-Handler-4:null) Executing: qemu-img
> info
> /mnt/b8311c72-fe75-3832-98fc-975445028a12/01ab129f-aaf6-4b1a-8e2a-093bee0b811c.raw
>
> DEBUG [utils.script.Script] (agentRequest-Handler-4:null) Execution is
> successful.
> DEBUG [utils.script.Script] (agentRequest-Handler-4:null) Executing:
> /bin/bash -c cp -f
> /mnt/b8311c72-fe75-3832-98fc-975445028a12/01ab129f-aaf6-4b1a-8e2a-093bee0b811c.raw
> /dev/STOREY1/1e1ae58c-ca3f-4d4a-b0fd-92d18d07fb7c
> {quote}
> Since the original format was actually QCOW2 (from the previous step) this
> lead to data loss once the volume is expunged from the previous pool and from
> the nfs storage!
> Migration worked before we migrated to 4.2.1.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
