[
https://issues.apache.org/jira/browse/CLOUDSTACK-6460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136210#comment-14136210
]
Simon Weller commented on CLOUDSTACK-6460:
------------------------------------------
I've been digging into this a lot today, and I think I've narrowed the problem
down to the attachVolume functionality.
It appears that during the attacheVolume operation, the volume format is being
set to QCOW2 in the database.
Steps to reproduce this:
1. Create a new volume either via GUI or API.
Database shows format field to be NULL.
2. Attach volume to VM.
At this point the format field is set to QCOW2, even though it's really RAW.
How's the database table data:
After Volume Creation:
mysql> select * from volumes where id='17451';
+-------+------------+-----------+---------+--------------+-------------+-----------+---------+--------------------------------------+------------+--------+------+--------+----------------+------------+---------+-------------+-----------+------------------+-------------+----------------------------+-------------+---------------------+----------+---------------------+---------+-----------+------------+--------------+-----------+------------------------+--------+----------------+--------+----------+----------+---------------+
| id | account_id | domain_id | pool_id | last_pool_id | instance_id |
device_id | name | uuid | size |
folder | path | pod_id | data_center_id | iscsi_name | host_ip | volume_type |
pool_type | disk_offering_id | template_id | first_snapshot_backup_uuid |
recreatable | created | attached | updated | removed |
state | chain_info | update_count | disk_type | vm_snapshot_chain_size |
iso_id | display_volume | format | min_iops | max_iops | hv_ss_reserve |
+-------+------------+-----------+---------+--------------+-------------+-----------+---------+--------------------------------------+------------+--------+------+--------+----------------+------------+---------+-------------+-----------+------------------+-------------+----------------------------+-------------+---------------------+----------+---------------------+---------+-----------+------------+--------------+-----------+------------------------+--------+----------------+--------+----------+----------+---------------+
| 17451 | 6 | 1 | NULL | NULL | NULL |
NULL | testnew | 9fc165e4-d383-4367-ab0a-9a55e5566110 | 5368709120 | NULL |
NULL | NULL | 2 | NULL | NULL | DATADISK | NULL
| 3 | NULL | NULL | 0 |
2014-09-16 20:14:25 | NULL | 2014-09-16 20:14:25 | NULL | Allocated |
NULL | 0 | NULL | NULL | NULL |
1 | NULL | NULL | NULL | NULL |
+-------+------------+-----------+---------+--------------+-------------+-----------+---------+--------------------------------------+------------+--------+------+--------+----------------+------------+---------+-------------+-----------+------------------+-------------+----------------------------+-------------+---------------------+----------+---------------------+---------+-----------+------------+--------------+-----------+------------------------+--------+----------------+--------+----------+----------+---------------+
1 row in set (0.00 sec)
mysql> select * from storage_pool;
+-----+-----------+--------------------------------------+-----------+------+----------------+--------+------------+---------------+----------------+--------------+-----------+------------+---------------------+---------+-------------+--------+-----------------------+---------+------------+---------+---------------+
| id | name | uuid | pool_type | port |
data_center_id | pod_id | cluster_id | used_bytes | capacity_bytes |
host_address | user_info | path | created | removed |
update_time | status | storage_provider_name | scope | hypervisor | managed |
capacity_iops |
+-----+-----------+--------------------------------------+-----------+------+----------------+--------+------------+---------------+----------------+--------------+-----------+------------+---------------------+---------+-------------+--------+-----------------------+---------+------------+---------+---------------+
| 201 | clvm1 | 076d1cd7-9c80-4302-8e13-ea8187a9a96b | CLVM | 0 |
2 | 2 | 2 | 1022336434176 | 1099507433472 | localhost
| NULL | /csstore01 | 2014-01-15 22:02:18 | NULL | NULL | Up
| DefaultPrimary | CLUSTER | NULL | 0 | NULL |
| 202 | csstore02 | 8f624e46-33fe-473c-9d05-62aebc151d1b | CLVM | 0 |
2 | 2 | 2 | 1019077459968 | 1099507433472 | localhost
| NULL | /csstore02 | 2014-01-16 19:17:47 | NULL | NULL | Up
| DefaultPrimary | CLUSTER | NULL | 0 | NULL |
+-----+-----------+--------------------------------------+-----------+------+----------------+--------+------------+---------------+----------------+--------------+-----------+------------+---------------------+---------+-------------+--------+-----------------------+---------+------------+---------+---------------+
After Volume Attach:
mysql> select * from volumes where id='17451';
+-------+------------+-----------+---------+--------------+-------------+-----------+---------+--------------------------------------+------------+--------+--------------------------------------+--------+----------------+------------+---------+-------------+-----------+------------------+-------------+----------------------------+-------------+---------------------+---------------------+---------------------+---------+-------+------------+--------------+-----------+------------------------+--------+----------------+--------+----------+----------+---------------+
| id | account_id | domain_id | pool_id | last_pool_id | instance_id |
device_id | name | uuid | size |
folder | path | pod_id | data_center_id |
iscsi_name | host_ip | volume_type | pool_type | disk_offering_id | template_id
| first_snapshot_backup_uuid | recreatable | created | attached
| updated | removed | state | chain_info | update_count |
disk_type | vm_snapshot_chain_size | iso_id | display_volume | format |
min_iops | max_iops | hv_ss_reserve |
+-------+------------+-----------+---------+--------------+-------------+-----------+---------+--------------------------------------+------------+--------+--------------------------------------+--------+----------------+------------+---------+-------------+-----------+------------------+-------------+----------------------------+-------------+---------------------+---------------------+---------------------+---------+-------+------------+--------------+-----------+------------------------+--------+----------------+--------+----------+----------+---------------+
| 17451 | 6 | 1 | 201 | NULL | 22370 |
1 | testnew | 9fc165e4-d383-4367-ab0a-9a55e5566110 | 5368709120 | NULL |
9fc165e4-d383-4367-ab0a-9a55e5566110 | NULL | 2 | NULL |
NULL | DATADISK | NULL | 3 | NULL | NULL
| 0 | 2014-09-16 20:14:25 | 2014-09-16 20:19:36 |
2014-09-16 20:19:36 | NULL | Ready | NULL | 2 | NULL |
NULL | NULL | 1 | QCOW2 | NULL | NULL
| NULL |
+-------+------------+-----------+---------+--------------+-------------+-----------+---------+--------------------------------------+------------+--------+--------------------------------------+--------+----------------+------------+---------+-------------+-----------+------------------+-------------+----------------------------+-------------+---------------------+---------------------+---------------------+---------+-------+------------+--------------+-----------+------------------------+--------+----------------+--------+----------+----------+---------------+
I'm still working on tracing how this method works in terms of sourcing it's
format. I'm going to take a stab in the dark here and say that it's possible
there is an assumption being made that all KVM hypervisor storage is QCOW2,
rather than the method looking at the primary storage type, which is obviously
independent from the hypervisor.
- Si
> Migration of CLVM volumes to another primary storage fail
> ---------------------------------------------------------
>
> Key: CLOUDSTACK-6460
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-6460
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: KVM, Volumes
> Affects Versions: 4.2.0, 4.2.1, 4.3.0, 4.4.0
> Environment: KVM clusters with fiber channel SAN storage, CLVM volumes
> Reporter: Salvatore Sciacco
> Attachments: CLOUDSTACK-6460-darrentang.patch,
> CLOUDSTACK-6460-darrentang_2.patch, cloudstack-6460.patch,
> cloudstack-6460_44.patch
>
>
> ACS version: 4.2.1
> Hypervisors: KVM
> Storage pool type: CLVM
> Since we upgraded from 4.1 to 4.2.1 moving volumes to a different primary
> storage pool fail. I've enabled debug on the agents side and I think there is
> a problem with the format type conversion
> Volume on database had format QCOW2
> these are the parameters for the first step (CLVM -> NFS):
> {quote}
> "srcTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":
> "uuid":"cda46430-52d7-4bf0-b0c2-adfc78dd011c","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":"uuid":"655d6965-b3f3-4118-a970-d50cf6afc365","id":211,"poolType":"CLVM","host":"localhost","path":"/FC10KY1","port":0,"name":"ROOT-4450","size":5368709120,"path":"39a25daf-23a1-4b65-99ac-fb98469ac197","volumeId":5937,"vmName":"i-402-4450-VM","accountId":402,"format":"QCOW2","id":5937,"hypervisorType":"KVM"}
> "destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"cda46430-52d7-4bf0-b0c2-adfc78dd011c","volumeType":"ROOT","dataStore":{"com.cloud.agent.api.to.NfsTO":
>
> "_url":"nfs://192.168.11.6/home/a1iwstack","_role":"Image"},"name":"ROOT-4450","size":5368709120,"path":"volumes/402/5937","volumeId":5937,"vmName":"i-402-4450-VM","accountId":402,"format":"QCOW2","id":5937,"hypervisorType":"KVM"}
> {quote}
> Those commads are translated into the agent:
> {quote}
> DEBUG [utils.script.Script] (agentRequest-Handler-1:null) Executing: qemu-img
> info /dev/FC10KY1/39a25daf-23a1-4b65-99ac-fb98469ac197
> DEBUG [utils.script.Script] (agentRequest-Handler-1:null) Execution is
> successful.
> DEBUG [utils.script.Script] (agentRequest-Handler-1:null) Executing:
> /bin/bash -c cp -f /dev/FC10KY1/39a25daf-23a1-4b65-99ac-fb98469ac197
> /mnt/b8311c72-fe75-3832-98fc-975445028a12/5c713376-c418-478c-8a31-89c4181cb48e.qcow2
>
> {quote}
> With the result that the output file isn't a qcow2 file but a raw partition,
> which in turn make the next step fail.
> (NFS -> CLVM)
> {quote}
> DEBUG [utils.script.Script] (agentRequest-Handler-2:) Executing: qemu-img
> info
> /mnt/b8311c72-fe75-3832-98fc-975445028a12/b9303d8d-cd51-4b6c-a244-43c405df4238.qcow2
>
> DEBUG [utils.script.Script] (agentRequest-Handler-2:) Execution is successful.
> DEBUG [utils.script.Script] (agentRequest-Handler-2:) Executing: qemu-img
> convert -f qcow2 -O raw
> /mnt/b8311c72-fe75-3832-98fc-975445028a12/b9303d8d-cd51-4b6c-a244-43c405df4238.qcow2
> /dev/FCSTORAGE/da162325-467b-4e78-af07-4bad85470d66
> DEBUG [utils.script.Script] (agentRequest-Handler-2:) Exit value is 1
> DEBUG [utils.script.Script] (agentRequest-Handler-2:) qemu-img: Could not
> open
> '/mnt/b8311c72-fe75-3832-98fc-975445028a12/b9303d8d-cd51-4b6c-a244-43c405df4238.qcow2'qemu-img:
> Could not open
> '/mnt/b8311c72-fe75-3832-98fc-975445028a12/b9303d8d-cd51-4b6c-a244-43c405df4238.qcow2'
> ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:) Failed to
> convert
> /mnt/b8311c72-fe75-3832-98fc-975445028a12/b9303d8d-cd51-4b6c-a244-43c405df4238.qcow2
> to /dev/FCSTORAGE/da162325-467b-4e78-af07-4bad85470d66 the error was:
> qemu-img: Could not open
> '/mnt/b8311c72-fe75-3832-98fc-975445028a12/b9303d8d-cd51-4b6c-a244-43c405df4238.qcow2'qemu-img:
> Could not open
> '/mnt/b8311c72-fe75-3832-98fc-975445028a12/b9303d8d-cd51-4b6c-a244-43c405df4238.qcow2'
> {quote}
> If I change on the database the format of the volume to RAW the effect is
> even worse as data is lost in the process!
> These are the parameter for the first step (CLVM => NFS)
> {quote}
> "srcTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"cda46430-52d7-4bf0-b0c2-adfc78dd011c","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"655d6965-b3f3-4118-a970d50cf6afc365","id":211,"poolType":"CLVM","host":"localhost","path":"/FC10KY1","port":0,"name":"ROOT-4450"
> ,"size":5368709120,"path":"39a25daf-23a1-4b65-99ac-fb98469ac197","volumeId":5937,"vmName":"i-4024450VM","accountId":402,"format":"RAW","id":5937,"hypervisorType":"KVM"
> "destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":"uuid":"cda46430-52d7-4bf0-b0c2-adfc78dd011c","volumeType":"ROOT","dataStore":{"com.cloud.agent.api.to.NfsTO":
>
> "_url":"nfs://192.168.11.6/home/a1iwstack","_role":"Image"}},"name":"ROOT4450","size":5368709120,"path":"volumes/402/5937","volumeId":5937,"vmName":"i-402-4450-VM","accountId":402,"format":"RAW","id":5937,"hypervisorType":"KVM"}
> {quote}
> this time the output is converted to qcow2!
> {quote}
> DEBUG [utils.script.Script] (agentRequest-Handler-3:null) Executing: qemu-img
> info /dev/FC10KY1/39a25daf-23a1-4b65-99ac-fb98469ac197
> DEBUG [utils.script.Script] (agentRequest-Handler-3:null) Execution is
> successful.
> DEBUG [utils.script.Script] (agentRequest-Handler-3:null) Executing: qemu-img
> convert -f raw -O qcow2 /dev/FC10KY1/39a25daf-23a1-4b65-99ac-fb98469ac197
> /mnt/b8311c72-fe75-3832-98fc-975445028a12/01ab129f-aaf6-4b1a-8e2a-093bee0b811c.raw
>
> {quote}
> and data is lost in the next step (NFS -> CLVM):
> {quote}
> "srcTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"cda46430-52d7-4bf0-b0c2-adfc78dd011c","volumeType":"ROOT","dataStore":{"com.cl
> oud.agent.api.to.NfsTO":{"_url":"nfs://192.168.11.6/home/a1iwstack","_role":"Image"}},"name":"ROOT4450","size":5368709120,"path":"volumes/402/5937/01ab129f-aaf6-4b1a-8e2a-093bee0b811c.raw","volumeId":5937,"vmName":"i-402-4450-VM","accountId":402,"format":"RAW","id":5937,"hypervisorType":"KVM"
> "destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"2d684689-efbc-4ce3-95ad-b38e80a1afcf","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4d59fde3-4c93-4aa5-8041-fab797b90899","id":214,"poolType":
> "CLVM","host":"localhost","path":"/STOREY1","port":0}},"name":"ROOT-4450","size":5368709120,"path":"39a25daf-23a1-4b65-99ac-fb98469ac197","volumeId":5967,"vmName":"i-402-4450-VM","accountId":402,"format":"RAW","id":5967,"hypervisorType":"KVM"
> DEBUG [utils.script.Script] (agentRequest-Handler-4:null) Executing: qemu-img
> info
> /mnt/b8311c72-fe75-3832-98fc-975445028a12/01ab129f-aaf6-4b1a-8e2a-093bee0b811c.raw
>
> DEBUG [utils.script.Script] (agentRequest-Handler-4:null) Execution is
> successful.
> DEBUG [utils.script.Script] (agentRequest-Handler-4:null) Executing:
> /bin/bash -c cp -f
> /mnt/b8311c72-fe75-3832-98fc-975445028a12/01ab129f-aaf6-4b1a-8e2a-093bee0b811c.raw
> /dev/STOREY1/1e1ae58c-ca3f-4d4a-b0fd-92d18d07fb7c
> {quote}
> Since the original format was actually QCOW2 (from the previous step) this
> lead to data loss once the volume is expunged from the previous pool and from
> the nfs storage!
> Migration worked before we migrated to 4.2.1.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
