[
https://issues.apache.org/jira/browse/CLOUDSTACK-5738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangeetha Hariharan updated CLOUDSTACK-5738:
--------------------------------------------
Description:
KVM - Parallel Snapshots - Few Snapshots are stuck in "Creating" state for more
than 30 minutes.
Set up:
Advanced zone set up with 2 KVM (rhel 63) hosts.
Steps to reproduce the problem:
Deploy about 17 Vms in both hosts.
Start "Hourly" snapshot for all the ROOT volumes of these Vms.
Few Snapshots are stuck in "Creating" state for more than 30 minutes now.
mysql> select id,volume_id,name,status,created from snapshots ;
+----+-----------+-------------------------------------------------+-----------+---------------------+
| id | volume_id | name | status
| created |
+----+-----------+-------------------------------------------------+-----------+---------------------+
| 1 | 3 | TestVM-1_ROOT-3_20131231011723 | Destroyed
| 2013-12-31 01:17:23 |
| 2 | 5 | TestVM-tiny-host-0ps-0-0_ROOT-5_20131231222951 | Destroyed
| 2013-12-31 22:29:51 |
| 3 | 10 | test-new_ROOT-10_20140102201054 | BackedUp
| 2014-01-02 20:10:54 |
| 4 | 22 | TestVM-tiny-host-0ps-0-6_ROOT-22_20140102234418 | BackedUp
| 2014-01-02 23:44:18 |
| 5 | 21 | TestVM-tiny-host-0ps-0-5_ROOT-21_20140102234418 | BackedUp
| 2014-01-02 23:44:18 |
| 6 | 20 | TestVM-tiny-host-0ps-0-4_ROOT-20_20140102234418 | BackedUp
| 2014-01-02 23:44:18 |
| 7 | 19 | TestVM-tiny-host-0ps-0-3_ROOT-19_20140102234418 | BackedUp
| 2014-01-02 23:44:18 |
| 8 | 18 | TestVM-tiny-host-0ps-0-2_ROOT-18_20140102234418 | BackedUp
| 2014-01-02 23:44:18 |
| 9 | 17 | TestVM-tiny-host-0ps-0-1_ROOT-17_20140102234418 | BackedUp
| 2014-01-02 23:44:18 |
| 10 | 16 | TestVM-tiny-host-0ps-0-0_ROOT-16_20140102234419 | BackedUp
| 2014-01-02 23:44:19 |
| 11 | 14 | TestVM-1_ROOT-14_20140102234419 | BackedUp
| 2014-01-02 23:44:19 |
| 12 | 31 | TestVM-tiny-host-1ps-0-5_ROOT-31_20140102234419 | BackedUp
| 2014-01-02 23:44:19 |
| 13 | 30 | TestVM-tiny-host-1ps-0-4_ROOT-30_20140102234419 | Error
| 2014-01-02 23:44:19 |
| 14 | 29 | TestVM-tiny-host-1ps-0-3_ROOT-29_20140102234419 | Error
| 2014-01-02 23:44:19 |
| 15 | 28 | TestVM-tiny-host-1ps-0-2_ROOT-28_20140102234419 | Error
| 2014-01-02 23:44:19 |
| 16 | 27 | TestVM-tiny-host-1ps-0-1_ROOT-27_20140102234419 | BackedUp
| 2014-01-02 23:44:19 |
| 17 | 26 | TestVM-tiny-host-1ps-0-0_ROOT-26_20140102234419 | BackedUp
| 2014-01-02 23:44:19 |
| 18 | 25 | TestVM-tiny-host-0ps-0-9_ROOT-25_20140102234419 | BackedUp
| 2014-01-02 23:44:19 |
| 19 | 24 | TestVM-tiny-host-0ps-0-8_ROOT-24_20140102234419 | BackedUp
| 2014-01-02 23:44:19 |
| 20 | 23 | TestVM-tiny-host-0ps-0-7_ROOT-23_20140102234419 | BackingUp
| 2014-01-02 23:44:19 |
| 21 | 22 | TestVM-tiny-host-0ps-0-6_ROOT-22_20140103014418 | BackingUp
| 2014-01-03 01:44:18 |
| 22 | 21 | TestVM-tiny-host-0ps-0-5_ROOT-21_20140103014418 | BackingUp
| 2014-01-03 01:44:18 |
| 23 | 20 | TestVM-tiny-host-0ps-0-4_ROOT-20_20140103014418 | BackingUp
| 2014-01-03 01:44:18 |
| 24 | 18 | TestVM-tiny-host-0ps-0-2_ROOT-18_20140103014418 | BackingUp
| 2014-01-03 01:44:18 |
| 25 | 17 | TestVM-tiny-host-0ps-0-1_ROOT-17_20140103014418 | BackingUp
| 2014-01-03 01:44:18 |
| 26 | 31 | TestVM-tiny-host-1ps-0-5_ROOT-31_20140103014418 | BackedUp
| 2014-01-03 01:44:18 |
| 27 | 30 | TestVM-tiny-host-1ps-0-4_ROOT-30_20140103014419 | BackedUp
| 2014-01-03 01:44:19 |
| 28 | 29 | TestVM-tiny-host-1ps-0-3_ROOT-29_20140103014419 | BackedUp
| 2014-01-03 01:44:19 |
| 29 | 28 | TestVM-tiny-host-1ps-0-2_ROOT-28_20140103014419 | Error
| 2014-01-03 01:44:19 |
| 30 | 27 | TestVM-tiny-host-1ps-0-1_ROOT-27_20140103014419 | BackedUp
| 2014-01-03 01:44:19 |
| 31 | 26 | TestVM-tiny-host-1ps-0-0_ROOT-26_20140103014419 | BackedUp
| 2014-01-03 01:44:19 |
| 32 | 19 | TestVM-tiny-host-0ps-0-3_ROOT-19_20140103020918 | Creating
| 2014-01-03 02:09:18 |
| 33 | 16 | TestVM-tiny-host-0ps-0-0_ROOT-16_20140103020918 | Creating
| 2014-01-03 02:09:18 |
| 34 | 14 | TestVM-1_ROOT-14_20140103020918 | Creating
| 2014-01-03 02:09:18 |
| 35 | 25 | TestVM-tiny-host-0ps-0-9_ROOT-25_20140103020918 | Creating
| 2014-01-03 02:09:18 |
| 36 | 24 | TestVM-tiny-host-0ps-0-8_ROOT-24_20140103020918 | Creating
| 2014-01-03 02:09:18 |
+----+-----------+-------------------------------------------------+-----------+---------------------+
was:
KVM - Parallel Snapshots - Few Snapshots are stuck in "Creating" state for more
than 30 minutes.
mysql> select id,volume_id,name,status,created from snapshots ;
+----+-----------+-------------------------------------------------+-----------+---------------------+
| id | volume_id | name | status
| created |
+----+-----------+-------------------------------------------------+-----------+---------------------+
| 1 | 3 | TestVM-1_ROOT-3_20131231011723 | Destroyed
| 2013-12-31 01:17:23 |
| 2 | 5 | TestVM-tiny-host-0ps-0-0_ROOT-5_20131231222951 | Destroyed
| 2013-12-31 22:29:51 |
| 3 | 10 | test-new_ROOT-10_20140102201054 | BackedUp
| 2014-01-02 20:10:54 |
| 4 | 22 | TestVM-tiny-host-0ps-0-6_ROOT-22_20140102234418 | BackedUp
| 2014-01-02 23:44:18 |
| 5 | 21 | TestVM-tiny-host-0ps-0-5_ROOT-21_20140102234418 | BackedUp
| 2014-01-02 23:44:18 |
| 6 | 20 | TestVM-tiny-host-0ps-0-4_ROOT-20_20140102234418 | BackedUp
| 2014-01-02 23:44:18 |
| 7 | 19 | TestVM-tiny-host-0ps-0-3_ROOT-19_20140102234418 | BackedUp
| 2014-01-02 23:44:18 |
| 8 | 18 | TestVM-tiny-host-0ps-0-2_ROOT-18_20140102234418 | BackedUp
| 2014-01-02 23:44:18 |
| 9 | 17 | TestVM-tiny-host-0ps-0-1_ROOT-17_20140102234418 | BackedUp
| 2014-01-02 23:44:18 |
| 10 | 16 | TestVM-tiny-host-0ps-0-0_ROOT-16_20140102234419 | BackedUp
| 2014-01-02 23:44:19 |
| 11 | 14 | TestVM-1_ROOT-14_20140102234419 | BackedUp
| 2014-01-02 23:44:19 |
| 12 | 31 | TestVM-tiny-host-1ps-0-5_ROOT-31_20140102234419 | BackedUp
| 2014-01-02 23:44:19 |
| 13 | 30 | TestVM-tiny-host-1ps-0-4_ROOT-30_20140102234419 | Error
| 2014-01-02 23:44:19 |
| 14 | 29 | TestVM-tiny-host-1ps-0-3_ROOT-29_20140102234419 | Error
| 2014-01-02 23:44:19 |
| 15 | 28 | TestVM-tiny-host-1ps-0-2_ROOT-28_20140102234419 | Error
| 2014-01-02 23:44:19 |
| 16 | 27 | TestVM-tiny-host-1ps-0-1_ROOT-27_20140102234419 | BackedUp
| 2014-01-02 23:44:19 |
| 17 | 26 | TestVM-tiny-host-1ps-0-0_ROOT-26_20140102234419 | BackedUp
| 2014-01-02 23:44:19 |
| 18 | 25 | TestVM-tiny-host-0ps-0-9_ROOT-25_20140102234419 | BackedUp
| 2014-01-02 23:44:19 |
| 19 | 24 | TestVM-tiny-host-0ps-0-8_ROOT-24_20140102234419 | BackedUp
| 2014-01-02 23:44:19 |
| 20 | 23 | TestVM-tiny-host-0ps-0-7_ROOT-23_20140102234419 | BackingUp
| 2014-01-02 23:44:19 |
| 21 | 22 | TestVM-tiny-host-0ps-0-6_ROOT-22_20140103014418 | BackingUp
| 2014-01-03 01:44:18 |
| 22 | 21 | TestVM-tiny-host-0ps-0-5_ROOT-21_20140103014418 | BackingUp
| 2014-01-03 01:44:18 |
| 23 | 20 | TestVM-tiny-host-0ps-0-4_ROOT-20_20140103014418 | BackingUp
| 2014-01-03 01:44:18 |
| 24 | 18 | TestVM-tiny-host-0ps-0-2_ROOT-18_20140103014418 | BackingUp
| 2014-01-03 01:44:18 |
| 25 | 17 | TestVM-tiny-host-0ps-0-1_ROOT-17_20140103014418 | BackingUp
| 2014-01-03 01:44:18 |
| 26 | 31 | TestVM-tiny-host-1ps-0-5_ROOT-31_20140103014418 | BackedUp
| 2014-01-03 01:44:18 |
| 27 | 30 | TestVM-tiny-host-1ps-0-4_ROOT-30_20140103014419 | BackedUp
| 2014-01-03 01:44:19 |
| 28 | 29 | TestVM-tiny-host-1ps-0-3_ROOT-29_20140103014419 | BackedUp
| 2014-01-03 01:44:19 |
| 29 | 28 | TestVM-tiny-host-1ps-0-2_ROOT-28_20140103014419 | Error
| 2014-01-03 01:44:19 |
| 30 | 27 | TestVM-tiny-host-1ps-0-1_ROOT-27_20140103014419 | BackedUp
| 2014-01-03 01:44:19 |
| 31 | 26 | TestVM-tiny-host-1ps-0-0_ROOT-26_20140103014419 | BackedUp
| 2014-01-03 01:44:19 |
| 32 | 19 | TestVM-tiny-host-0ps-0-3_ROOT-19_20140103020918 | Creating
| 2014-01-03 02:09:18 |
| 33 | 16 | TestVM-tiny-host-0ps-0-0_ROOT-16_20140103020918 | Creating
| 2014-01-03 02:09:18 |
| 34 | 14 | TestVM-1_ROOT-14_20140103020918 | Creating
| 2014-01-03 02:09:18 |
| 35 | 25 | TestVM-tiny-host-0ps-0-9_ROOT-25_20140103020918 | Creating
| 2014-01-03 02:09:18 |
| 36 | 24 | TestVM-tiny-host-0ps-0-8_ROOT-24_20140103020918 | Creating
| 2014-01-03 02:09:18 |
+----+-----------+-------------------------------------------------+-----------+---------------------+
> KVM - Parallel Snapshots - Few Snapshots are stuck in "Creating" state for
> more than 30 minutes.
> ------------------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-5738
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5738
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Management Server
> Affects Versions: 4.3.0
> Environment: Build from 4.3
> Reporter: Sangeetha Hariharan
> Fix For: 4.3.0
>
> Attachments: kvm.rar, kvm.rar
>
>
> KVM - Parallel Snapshots - Few Snapshots are stuck in "Creating" state for
> more than 30 minutes.
> Set up:
> Advanced zone set up with 2 KVM (rhel 63) hosts.
> Steps to reproduce the problem:
> Deploy about 17 Vms in both hosts.
> Start "Hourly" snapshot for all the ROOT volumes of these Vms.
> Few Snapshots are stuck in "Creating" state for more than 30 minutes now.
> mysql> select id,volume_id,name,status,created from snapshots ;
> +----+-----------+-------------------------------------------------+-----------+---------------------+
> | id | volume_id | name | status
> | created |
> +----+-----------+-------------------------------------------------+-----------+---------------------+
> | 1 | 3 | TestVM-1_ROOT-3_20131231011723 |
> Destroyed | 2013-12-31 01:17:23 |
> | 2 | 5 | TestVM-tiny-host-0ps-0-0_ROOT-5_20131231222951 |
> Destroyed | 2013-12-31 22:29:51 |
> | 3 | 10 | test-new_ROOT-10_20140102201054 | BackedUp
> | 2014-01-02 20:10:54 |
> | 4 | 22 | TestVM-tiny-host-0ps-0-6_ROOT-22_20140102234418 | BackedUp
> | 2014-01-02 23:44:18 |
> | 5 | 21 | TestVM-tiny-host-0ps-0-5_ROOT-21_20140102234418 | BackedUp
> | 2014-01-02 23:44:18 |
> | 6 | 20 | TestVM-tiny-host-0ps-0-4_ROOT-20_20140102234418 | BackedUp
> | 2014-01-02 23:44:18 |
> | 7 | 19 | TestVM-tiny-host-0ps-0-3_ROOT-19_20140102234418 | BackedUp
> | 2014-01-02 23:44:18 |
> | 8 | 18 | TestVM-tiny-host-0ps-0-2_ROOT-18_20140102234418 | BackedUp
> | 2014-01-02 23:44:18 |
> | 9 | 17 | TestVM-tiny-host-0ps-0-1_ROOT-17_20140102234418 | BackedUp
> | 2014-01-02 23:44:18 |
> | 10 | 16 | TestVM-tiny-host-0ps-0-0_ROOT-16_20140102234419 | BackedUp
> | 2014-01-02 23:44:19 |
> | 11 | 14 | TestVM-1_ROOT-14_20140102234419 | BackedUp
> | 2014-01-02 23:44:19 |
> | 12 | 31 | TestVM-tiny-host-1ps-0-5_ROOT-31_20140102234419 | BackedUp
> | 2014-01-02 23:44:19 |
> | 13 | 30 | TestVM-tiny-host-1ps-0-4_ROOT-30_20140102234419 | Error
> | 2014-01-02 23:44:19 |
> | 14 | 29 | TestVM-tiny-host-1ps-0-3_ROOT-29_20140102234419 | Error
> | 2014-01-02 23:44:19 |
> | 15 | 28 | TestVM-tiny-host-1ps-0-2_ROOT-28_20140102234419 | Error
> | 2014-01-02 23:44:19 |
> | 16 | 27 | TestVM-tiny-host-1ps-0-1_ROOT-27_20140102234419 | BackedUp
> | 2014-01-02 23:44:19 |
> | 17 | 26 | TestVM-tiny-host-1ps-0-0_ROOT-26_20140102234419 | BackedUp
> | 2014-01-02 23:44:19 |
> | 18 | 25 | TestVM-tiny-host-0ps-0-9_ROOT-25_20140102234419 | BackedUp
> | 2014-01-02 23:44:19 |
> | 19 | 24 | TestVM-tiny-host-0ps-0-8_ROOT-24_20140102234419 | BackedUp
> | 2014-01-02 23:44:19 |
> | 20 | 23 | TestVM-tiny-host-0ps-0-7_ROOT-23_20140102234419 |
> BackingUp | 2014-01-02 23:44:19 |
> | 21 | 22 | TestVM-tiny-host-0ps-0-6_ROOT-22_20140103014418 |
> BackingUp | 2014-01-03 01:44:18 |
> | 22 | 21 | TestVM-tiny-host-0ps-0-5_ROOT-21_20140103014418 |
> BackingUp | 2014-01-03 01:44:18 |
> | 23 | 20 | TestVM-tiny-host-0ps-0-4_ROOT-20_20140103014418 |
> BackingUp | 2014-01-03 01:44:18 |
> | 24 | 18 | TestVM-tiny-host-0ps-0-2_ROOT-18_20140103014418 |
> BackingUp | 2014-01-03 01:44:18 |
> | 25 | 17 | TestVM-tiny-host-0ps-0-1_ROOT-17_20140103014418 |
> BackingUp | 2014-01-03 01:44:18 |
> | 26 | 31 | TestVM-tiny-host-1ps-0-5_ROOT-31_20140103014418 | BackedUp
> | 2014-01-03 01:44:18 |
> | 27 | 30 | TestVM-tiny-host-1ps-0-4_ROOT-30_20140103014419 | BackedUp
> | 2014-01-03 01:44:19 |
> | 28 | 29 | TestVM-tiny-host-1ps-0-3_ROOT-29_20140103014419 | BackedUp
> | 2014-01-03 01:44:19 |
> | 29 | 28 | TestVM-tiny-host-1ps-0-2_ROOT-28_20140103014419 | Error
> | 2014-01-03 01:44:19 |
> | 30 | 27 | TestVM-tiny-host-1ps-0-1_ROOT-27_20140103014419 | BackedUp
> | 2014-01-03 01:44:19 |
> | 31 | 26 | TestVM-tiny-host-1ps-0-0_ROOT-26_20140103014419 | BackedUp
> | 2014-01-03 01:44:19 |
> | 32 | 19 | TestVM-tiny-host-0ps-0-3_ROOT-19_20140103020918 | Creating
> | 2014-01-03 02:09:18 |
> | 33 | 16 | TestVM-tiny-host-0ps-0-0_ROOT-16_20140103020918 | Creating
> | 2014-01-03 02:09:18 |
> | 34 | 14 | TestVM-1_ROOT-14_20140103020918 | Creating
> | 2014-01-03 02:09:18 |
> | 35 | 25 | TestVM-tiny-host-0ps-0-9_ROOT-25_20140103020918 | Creating
> | 2014-01-03 02:09:18 |
> | 36 | 24 | TestVM-tiny-host-0ps-0-8_ROOT-24_20140103020918 | Creating
> | 2014-01-03 02:09:18 |
> +----+-----------+-------------------------------------------------+-----------+---------------------+
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
