Dear all I experienced volume quota problem recently. DATA volumes in 'destroy' state seem to be counted in as volume usage, then I have to set a higher value to volume quota if I can't expunge those. If you know how to resolve it, could you help me?
Environment is as below. CloudStack: 4.3.0 HyperVisor: KVM OS: Management Server: CentOS 6.5 Host: CentOS 6.5 Primary Storage: Local Storage Secondary Storage: S3 Zone: Type: Advanced Pod in zone: 1 Cluster in pod: 9 Hosts per cluster: 16 I describe a failure situation in detail. At first, I encountered a VM deployment failure like this. 1. Try to deploy VM with a DATA volume. 2. Trye to start VM at host A. 3. VM exits by an error at host A. 4. Fail to start VM at host A. 5. Try to start at another host. 6. Fail to migrate another host because primary storage is local storage. 7. Fail to deploy VM. This failure was resolved by making host A maintainance mode and VM ROOT volume was expunged correctly. However, the Data volume entries still remain in database in 'destroy' state. Here are the ones. mysql> select * from cloud.volumes where account_id = 3 and state != "Expunged" and state != "Ready" \G; *************************** 1. row *************************** id: 543 account_id: 3 domain_id: 2 pool_id: 113 last_pool_id: NULL instance_id: NULL device_id: NULL name: DATA-282 uuid: 85e40f8e-5df5-4a58-91c2-ebc768f78213 size: 483183820800 folder: NULL path: 85e40f8e-5df5-4a58-91c2-ebc768f78213 pod_id: NULL data_center_id: 1 iscsi_name: NULL host_ip: NULL volume_type: DATADISK pool_type: NULL disk_offering_id: 15 template_id: NULL first_snapshot_backup_uuid: NULL recreatable: 0 created: 2014-05-22 16:22:32 attached: NULL updated: 2014-05-25 13:51:06 removed: NULL state: Destroy chain_info: NULL update_count: 10 disk_type: NULL vm_snapshot_chain_size: NULL iso_id: NULL display_volume: 1 format: QCOW2 min_iops: NULL max_iops: NULL hv_ss_reserve: NULL *************************** 2. row *************************** id: 549 account_id: 3 domain_id: 2 pool_id: 113 last_pool_id: NULL instance_id: NULL device_id: NULL name: DATA-285 uuid: 5e8050fc-56be-488f-a7bc-0262e1dd535d size: 483183820800 folder: NULL path: 5e8050fc-56be-488f-a7bc-0262e1dd535d pod_id: NULL data_center_id: 1 iscsi_name: NULL host_ip: NULL volume_type: DATADISK pool_type: NULL disk_offering_id: 15 template_id: NULL first_snapshot_backup_uuid: NULL recreatable: 0 created: 2014-05-23 02:36:46 attached: NULL updated: 2014-05-25 13:51:06 removed: NULL state: Destroy chain_info: NULL update_count: 10 disk_type: NULL vm_snapshot_chain_size: NULL iso_id: NULL display_volume: 1 format: QCOW2 min_iops: NULL max_iops: NULL hv_ss_reserve: NULL *************************** 3. row *************************** id: 553 account_id: 3 domain_id: 2 pool_id: 113 last_pool_id: NULL instance_id: NULL device_id: NULL name: DATA-287 uuid: 0505ad80-db90-4876-af60-b4d84efea3f5 size: 483183820800 folder: NULL path: 0505ad80-db90-4876-af60-b4d84efea3f5 pod_id: NULL data_center_id: 1 iscsi_name: NULL host_ip: NULL volume_type: DATADISK pool_type: NULL disk_offering_id: 15 template_id: NULL first_snapshot_backup_uuid: NULL recreatable: 0 created: 2014-05-23 02:49:45 attached: NULL updated: 2014-05-25 13:51:06 removed: NULL state: Destroy chain_info: NULL update_count: 10 disk_type: NULL vm_snapshot_chain_size: NULL iso_id: NULL display_volume: 1 format: QCOW2 min_iops: NULL max_iops: NULL hv_ss_reserve: NULL *************************** 4. row *************************** id: 555 account_id: 3 domain_id: 2 pool_id: 113 last_pool_id: NULL instance_id: NULL device_id: NULL name: DATA-288 uuid: 5aacc130-f6b0-4f9a-9660-f746b0d3d8aa size: 483183820800 folder: NULL path: 5aacc130-f6b0-4f9a-9660-f746b0d3d8aa pod_id: NULL data_center_id: 1 iscsi_name: NULL host_ip: NULL volume_type: DATADISK pool_type: NULL disk_offering_id: 15 template_id: NULL first_snapshot_backup_uuid: NULL recreatable: 0 created: 2014-05-23 03:00:14 attached: NULL updated: 2014-05-25 13:51:06 removed: NULL state: Destroy chain_info: NULL update_count: 10 disk_type: NULL vm_snapshot_chain_size: NULL iso_id: NULL display_volume: 1 format: QCOW2 min_iops: NULL max_iops: NULL hv_ss_reserve: NULL *************************** 5. row *************************** id: 557 account_id: 3 domain_id: 2 pool_id: 113 last_pool_id: NULL instance_id: NULL device_id: NULL name: DATA-290 uuid: 858970fa-ef83-40cf-9dc0-8db7b7fd47af size: 483183820800 folder: NULL path: 858970fa-ef83-40cf-9dc0-8db7b7fd47af pod_id: NULL data_center_id: 1 iscsi_name: NULL host_ip: NULL volume_type: DATADISK pool_type: NULL disk_offering_id: 15 template_id: NULL first_snapshot_backup_uuid: NULL recreatable: 0 created: 2014-05-23 03:02:17 attached: NULL updated: 2014-05-25 13:51:06 removed: NULL state: Destroy chain_info: NULL update_count: 10 disk_type: NULL vm_snapshot_chain_size: NULL iso_id: NULL display_volume: 1 format: QCOW2 min_iops: NULL max_iops: NULL hv_ss_reserve: NULL *************************** 6. row *************************** id: 559 account_id: 3 domain_id: 2 pool_id: 113 last_pool_id: NULL instance_id: NULL device_id: NULL name: DATA-291 uuid: a7a18d35-98ac-40cc-bd17-27a43a9d487f size: 483183820800 folder: NULL path: a7a18d35-98ac-40cc-bd17-27a43a9d487f pod_id: NULL data_center_id: 1 iscsi_name: NULL host_ip: NULL volume_type: DATADISK pool_type: NULL disk_offering_id: 15 template_id: NULL first_snapshot_backup_uuid: NULL recreatable: 0 created: 2014-05-23 03:07:43 attached: NULL updated: 2014-05-25 13:51:07 removed: NULL state: Destroy chain_info: NULL update_count: 10 disk_type: NULL vm_snapshot_chain_size: NULL iso_id: NULL display_volume: 1 format: QCOW2 min_iops: NULL max_iops: NULL hv_ss_reserve: NULL *************************** 7. row *************************** id: 561 account_id: 3 domain_id: 2 pool_id: 113 last_pool_id: NULL instance_id: NULL device_id: NULL name: DATA-292 uuid: f3687609-c722-49d8-b946-fb8589660554 size: 483183820800 folder: NULL path: f3687609-c722-49d8-b946-fb8589660554 pod_id: NULL data_center_id: 1 iscsi_name: NULL host_ip: NULL volume_type: DATADISK pool_type: NULL disk_offering_id: 15 template_id: NULL first_snapshot_backup_uuid: NULL recreatable: 0 created: 2014-05-23 03:14:56 attached: NULL updated: 2014-05-25 13:51:07 removed: NULL state: Destroy chain_info: NULL update_count: 10 disk_type: NULL vm_snapshot_chain_size: NULL iso_id: NULL display_volume: 1 format: QCOW2 min_iops: NULL max_iops: NULL hv_ss_reserve: NULL *************************** 8. row *************************** id: 563 account_id: 3 domain_id: 2 pool_id: 113 last_pool_id: NULL instance_id: NULL device_id: NULL name: DATA-293 uuid: 6e717388-1cf9-4ae9-b5f2-97316e8b6ae8 size: 483183820800 folder: NULL path: 6e717388-1cf9-4ae9-b5f2-97316e8b6ae8 pod_id: NULL data_center_id: 1 iscsi_name: NULL host_ip: NULL volume_type: DATADISK pool_type: NULL disk_offering_id: 15 template_id: NULL first_snapshot_backup_uuid: NULL recreatable: 0 created: 2014-05-23 03:19:10 attached: NULL updated: 2014-05-25 13:51:07 removed: NULL state: Destroy chain_info: NULL update_count: 10 disk_type: NULL vm_snapshot_chain_size: NULL iso_id: NULL display_volume: 1 format: QCOW2 min_iops: NULL max_iops: NULL hv_ss_reserve: NULL *************************** 9. row *************************** id: 565 account_id: 3 domain_id: 2 pool_id: 113 last_pool_id: NULL instance_id: NULL device_id: NULL name: DATA-294 uuid: 711277ec-38d0-4ee8-8828-6a92baf6083c size: 483183820800 folder: NULL path: 711277ec-38d0-4ee8-8828-6a92baf6083c pod_id: NULL data_center_id: 1 iscsi_name: NULL host_ip: NULL volume_type: DATADISK pool_type: NULL disk_offering_id: 15 template_id: NULL first_snapshot_backup_uuid: NULL recreatable: 0 created: 2014-05-23 03:26:11 attached: NULL updated: 2014-05-25 13:51:07 removed: NULL state: Destroy chain_info: NULL update_count: 10 disk_type: NULL vm_snapshot_chain_size: NULL iso_id: NULL display_volume: 1 format: QCOW2 min_iops: NULL max_iops: NULL hv_ss_reserve: NULL *************************** 10. row *************************** id: 567 account_id: 3 domain_id: 2 pool_id: 113 last_pool_id: NULL instance_id: NULL device_id: NULL name: DATA-295 uuid: 414a382c-e95b-44e2-b99f-ec5ede77f3f7 size: 483183820800 folder: NULL path: 414a382c-e95b-44e2-b99f-ec5ede77f3f7 pod_id: NULL data_center_id: 1 iscsi_name: NULL host_ip: NULL volume_type: DATADISK pool_type: NULL disk_offering_id: 15 template_id: NULL first_snapshot_backup_uuid: NULL recreatable: 0 created: 2014-05-23 04:01:11 attached: NULL updated: 2014-05-25 13:51:07 removed: NULL state: Destroy chain_info: NULL update_count: 10 disk_type: NULL vm_snapshot_chain_size: NULL iso_id: NULL display_volume: 1 format: QCOW2 min_iops: NULL max_iops: NULL hv_ss_reserve: NULL *************************** 11. row *************************** id: 569 account_id: 3 domain_id: 2 pool_id: 113 last_pool_id: NULL instance_id: NULL device_id: NULL name: DATA-296 uuid: 97b8fa8a-03da-4ef5-b643-d5e5f91aadf7 size: 483183820800 folder: NULL path: 97b8fa8a-03da-4ef5-b643-d5e5f91aadf7 pod_id: NULL data_center_id: 1 iscsi_name: NULL host_ip: NULL volume_type: DATADISK pool_type: NULL disk_offering_id: 15 template_id: NULL first_snapshot_backup_uuid: NULL recreatable: 0 created: 2014-05-23 04:04:59 attached: NULL updated: 2014-05-25 13:51:07 removed: NULL state: Destroy chain_info: NULL update_count: 10 disk_type: NULL vm_snapshot_chain_size: NULL iso_id: NULL display_volume: 1 format: QCOW2 min_iops: NULL max_iops: NULL hv_ss_reserve: NULL *************************** 12. row *************************** id: 571 account_id: 3 domain_id: 2 pool_id: 113 last_pool_id: NULL instance_id: NULL device_id: NULL name: DATA-298 uuid: f8b2038e-d607-4d7f-91ff-e90e5aae2485 size: 483183820800 folder: NULL path: f8b2038e-d607-4d7f-91ff-e90e5aae2485 pod_id: NULL data_center_id: 1 iscsi_name: NULL host_ip: NULL volume_type: DATADISK pool_type: NULL disk_offering_id: 15 template_id: NULL first_snapshot_backup_uuid: NULL recreatable: 0 created: 2014-05-23 04:06:12 attached: NULL updated: 2014-05-25 13:51:07 removed: NULL state: Destroy chain_info: NULL update_count: 10 disk_type: NULL vm_snapshot_chain_size: NULL iso_id: NULL display_volume: 1 format: QCOW2 min_iops: NULL max_iops: NULL hv_ss_reserve: NULL *************************** 13. row *************************** id: 573 account_id: 3 domain_id: 2 pool_id: 113 last_pool_id: NULL instance_id: NULL device_id: NULL name: DATA-299 uuid: 5612d2fe-1308-4fb0-9bf0-5653ee7d61fc size: 483183820800 folder: NULL path: 5612d2fe-1308-4fb0-9bf0-5653ee7d61fc pod_id: NULL data_center_id: 1 iscsi_name: NULL host_ip: NULL volume_type: DATADISK pool_type: NULL disk_offering_id: 15 template_id: NULL first_snapshot_backup_uuid: NULL recreatable: 0 created: 2014-05-23 05:03:27 attached: NULL updated: 2014-05-25 13:51:07 removed: NULL state: Destroy chain_info: NULL update_count: 10 disk_type: NULL vm_snapshot_chain_size: NULL iso_id: NULL display_volume: 1 format: QCOW2 min_iops: NULL max_iops: NULL hv_ss_reserve: NULL *************************** 14. row *************************** id: 575 account_id: 3 domain_id: 2 pool_id: 113 last_pool_id: NULL instance_id: NULL device_id: NULL name: DATA-300 uuid: a810d611-ecf4-4197-9bc7-8260bc05a698 size: 483183820800 folder: NULL path: a810d611-ecf4-4197-9bc7-8260bc05a698 pod_id: NULL data_center_id: 1 iscsi_name: NULL host_ip: NULL volume_type: DATADISK pool_type: NULL disk_offering_id: 15 template_id: NULL first_snapshot_backup_uuid: NULL recreatable: 0 created: 2014-05-23 05:05:53 attached: NULL updated: 2014-05-25 13:51:07 removed: NULL state: Destroy chain_info: NULL update_count: 10 disk_type: NULL vm_snapshot_chain_size: NULL iso_id: NULL display_volume: 1 format: QCOW2 min_iops: NULL max_iops: NULL hv_ss_reserve: NULL *************************** 15. row *************************** id: 577 account_id: 3 domain_id: 2 pool_id: 113 last_pool_id: NULL instance_id: NULL device_id: NULL name: DATA-302 uuid: f0d0e7d7-0c61-42e2-95f2-ea2f48a282f8 size: 483183820800 folder: NULL path: f0d0e7d7-0c61-42e2-95f2-ea2f48a282f8 pod_id: NULL data_center_id: 1 iscsi_name: NULL host_ip: NULL volume_type: DATADISK pool_type: NULL disk_offering_id: 15 template_id: NULL first_snapshot_backup_uuid: NULL recreatable: 0 created: 2014-05-23 05:06:55 attached: NULL updated: 2014-05-25 13:51:07 removed: NULL state: Destroy chain_info: NULL update_count: 10 disk_type: NULL vm_snapshot_chain_size: NULL iso_id: NULL display_volume: 1 format: QCOW2 min_iops: NULL max_iops: NULL hv_ss_reserve: NULL *************************** 16. row *************************** id: 579 account_id: 3 domain_id: 2 pool_id: 113 last_pool_id: NULL instance_id: NULL device_id: NULL name: DATA-303 uuid: 2d4b5116-6cf9-4e6a-9d0d-40c55d1a7113 size: 483183820800 folder: NULL path: 2d4b5116-6cf9-4e6a-9d0d-40c55d1a7113 pod_id: NULL data_center_id: 1 iscsi_name: NULL host_ip: NULL volume_type: DATADISK pool_type: NULL disk_offering_id: 15 template_id: NULL first_snapshot_backup_uuid: NULL recreatable: 0 created: 2014-05-23 05:09:58 attached: NULL updated: 2014-05-25 13:51:07 removed: NULL state: Destroy chain_info: NULL update_count: 10 disk_type: NULL vm_snapshot_chain_size: NULL iso_id: NULL display_volume: 1 format: QCOW2 min_iops: NULL max_iops: NULL hv_ss_reserve: NULL *************************** 17. row *************************** id: 581 account_id: 3 domain_id: 2 pool_id: 113 last_pool_id: NULL instance_id: NULL device_id: NULL name: DATA-304 uuid: 7e9d5f33-aeb5-496e-b505-4d70a3f44639 size: 483183820800 folder: NULL path: 7e9d5f33-aeb5-496e-b505-4d70a3f44639 pod_id: NULL data_center_id: 1 iscsi_name: NULL host_ip: NULL volume_type: DATADISK pool_type: NULL disk_offering_id: 15 template_id: NULL first_snapshot_backup_uuid: NULL recreatable: 0 created: 2014-05-23 05:10:35 attached: NULL updated: 2014-05-25 13:51:07 removed: NULL state: Destroy chain_info: NULL update_count: 10 disk_type: NULL vm_snapshot_chain_size: NULL iso_id: NULL display_volume: 1 format: QCOW2 min_iops: NULL max_iops: NULL hv_ss_reserve: NULL *************************** 18. row *************************** id: 583 account_id: 3 domain_id: 2 pool_id: 113 last_pool_id: NULL instance_id: NULL device_id: NULL name: DATA-305 uuid: 931547d0-db8a-42a0-9996-63d28aa7309d size: 483183820800 folder: NULL path: 931547d0-db8a-42a0-9996-63d28aa7309d pod_id: NULL data_center_id: 1 iscsi_name: NULL host_ip: NULL volume_type: DATADISK pool_type: NULL disk_offering_id: 15 template_id: NULL first_snapshot_backup_uuid: NULL recreatable: 0 created: 2014-05-23 05:14:18 attached: NULL updated: 2014-05-25 13:51:07 removed: NULL state: Destroy chain_info: NULL update_count: 10 disk_type: NULL vm_snapshot_chain_size: NULL iso_id: NULL display_volume: 1 format: QCOW2 min_iops: NULL max_iops: NULL hv_ss_reserve: NULL *************************** 19. row *************************** id: 585 account_id: 3 domain_id: 2 pool_id: 113 last_pool_id: NULL instance_id: NULL device_id: NULL name: DATA-306 uuid: aa3e78b3-f0a6-4463-bd79-91c12c333387 size: 483183820800 folder: NULL path: aa3e78b3-f0a6-4463-bd79-91c12c333387 pod_id: NULL data_center_id: 1 iscsi_name: NULL host_ip: NULL volume_type: DATADISK pool_type: NULL disk_offering_id: 15 template_id: NULL first_snapshot_backup_uuid: NULL recreatable: 0 created: 2014-05-23 05:17:25 attached: NULL updated: 2014-05-25 13:51:07 removed: NULL state: Destroy chain_info: NULL update_count: 10 disk_type: NULL vm_snapshot_chain_size: NULL iso_id: NULL display_volume: 1 format: QCOW2 min_iops: NULL max_iops: NULL hv_ss_reserve: NULL *************************** 20. row *************************** id: 587 account_id: 3 domain_id: 2 pool_id: 113 last_pool_id: NULL instance_id: NULL device_id: NULL name: DATA-308 uuid: 5108d1a3-e5b1-4dbe-8778-2be790510093 size: 483183820800 folder: NULL path: 5108d1a3-e5b1-4dbe-8778-2be790510093 pod_id: NULL data_center_id: 1 iscsi_name: NULL host_ip: NULL volume_type: DATADISK pool_type: NULL disk_offering_id: 15 template_id: NULL first_snapshot_backup_uuid: NULL recreatable: 0 created: 2014-05-23 05:19:18 attached: NULL updated: 2014-05-25 13:51:07 removed: NULL state: Destroy chain_info: NULL update_count: 10 disk_type: NULL vm_snapshot_chain_size: NULL iso_id: NULL display_volume: 1 format: QCOW2 min_iops: NULL max_iops: NULL hv_ss_reserve: NULL *************************** 21. row *************************** id: 589 account_id: 3 domain_id: 2 pool_id: 113 last_pool_id: NULL instance_id: NULL device_id: NULL name: DATA-309 uuid: 6da0244e-daed-45cc-aecc-c3e3482c1dd0 size: 483183820800 folder: NULL path: 6da0244e-daed-45cc-aecc-c3e3482c1dd0 pod_id: NULL data_center_id: 1 iscsi_name: NULL host_ip: NULL volume_type: DATADISK pool_type: NULL disk_offering_id: 15 template_id: NULL first_snapshot_backup_uuid: NULL recreatable: 0 created: 2014-05-23 05:22:00 attached: NULL updated: 2014-05-25 13:51:07 removed: NULL state: Destroy chain_info: NULL update_count: 10 disk_type: NULL vm_snapshot_chain_size: NULL iso_id: NULL display_volume: 1 format: QCOW2 min_iops: NULL max_iops: NULL hv_ss_reserve: NULL *************************** 22. row *************************** id: 591 account_id: 3 domain_id: 2 pool_id: 113 last_pool_id: NULL instance_id: NULL device_id: NULL name: DATA-311 uuid: 34692f00-8ea4-4e4a-91b4-8bf1f0ead0b3 size: 483183820800 folder: NULL path: 34692f00-8ea4-4e4a-91b4-8bf1f0ead0b3 pod_id: NULL data_center_id: 1 iscsi_name: NULL host_ip: NULL volume_type: DATADISK pool_type: NULL disk_offering_id: 15 template_id: NULL first_snapshot_backup_uuid: NULL recreatable: 0 created: 2014-05-23 05:23:12 attached: NULL updated: 2014-05-25 13:51:07 removed: NULL state: Destroy chain_info: NULL update_count: 10 disk_type: NULL vm_snapshot_chain_size: NULL iso_id: NULL display_volume: 1 format: QCOW2 min_iops: NULL max_iops: NULL hv_ss_reserve: NULL *************************** 23. row *************************** id: 594 account_id: 3 domain_id: 2 pool_id: 113 last_pool_id: NULL instance_id: NULL device_id: NULL name: DATA-313 uuid: 0fa95aa3-7582-47d7-b35c-9b178725a9d8 size: 483183820800 folder: NULL path: 0fa95aa3-7582-47d7-b35c-9b178725a9d8 pod_id: NULL data_center_id: 1 iscsi_name: NULL host_ip: NULL volume_type: DATADISK pool_type: NULL disk_offering_id: 15 template_id: NULL first_snapshot_backup_uuid: NULL recreatable: 0 created: 2014-05-23 05:53:03 attached: NULL updated: 2014-05-25 13:51:07 removed: NULL state: Destroy chain_info: NULL update_count: 10 disk_type: NULL vm_snapshot_chain_size: NULL iso_id: NULL display_volume: 1 format: QCOW2 min_iops: NULL max_iops: NULL hv_ss_reserve: NULL *************************** 24. row *************************** id: 596 account_id: 3 domain_id: 2 pool_id: 113 last_pool_id: NULL instance_id: NULL device_id: NULL name: DATA-314 uuid: 66089e18-6d16-42ad-be79-f251c1edd2ea size: 483183820800 folder: NULL path: 66089e18-6d16-42ad-be79-f251c1edd2ea pod_id: NULL data_center_id: 1 iscsi_name: NULL host_ip: NULL volume_type: DATADISK pool_type: NULL disk_offering_id: 15 template_id: NULL first_snapshot_backup_uuid: NULL recreatable: 0 created: 2014-05-23 05:55:50 attached: NULL updated: 2014-05-25 13:51:07 removed: NULL state: Destroy chain_info: NULL update_count: 10 disk_type: NULL vm_snapshot_chain_size: NULL iso_id: NULL display_volume: 1 format: QCOW2 min_iops: NULL max_iops: NULL hv_ss_reserve: NULL 24 rows in set (0.00 sec) ERROR: No query specified mysql> The incorrect DATA volume state is not a critical problem but bothers me with quota management about each account and domain. So, please let me know if you have a way to resolve this issue. Best Regards -- Hiroki Ohashi