[
https://issues.apache.org/jira/browse/CLOUDSTACK-5446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangeetha Hariharan reopened CLOUDSTACK-5446:
---------------------------------------------
Tested with build from 4.3.
When snapshots are created when NFS store is down , we now mark the snapshots
in "Error" state.
Bring up NFS server.
Attempting another snapshot on the same volume succeeds now.
Snapshots created on the primary store when NFS server is down , is not being
cleaned up.
These are the 2 entries from the snapshot_store_ref for the failed snapshot
attempts when NSF secondary store was down:
| 620 | 1 | 336 | 2014-01-03 23:09:38 | NULL | NULL |
Primary | 0 | 0 | 335 |
/mnt/c20ea198-e8ca-33c3-9f11-e361ec9b5532/a90e68e2-6b9c-4ddf-9435-746dffe030de/273939cf-0515-41c4-8553-5990b2770e2b
| Ready | 2 | 0 | 2014-01-03 23:09:39 | 33 |
| 622 | 1 | 337 | 2014-01-03 23:24:50 | NULL | NULL |
Primary | 0 | 0 | 336 |
/mnt/c20ea198-e8ca-33c3-9f11-e361ec9b5532/a90e68e2-6b9c-4ddf-9435-746dffe030de/c505ff5b-c4eb-4ad2-914b-9144c1ea5da2
| Ready | 2 | 0 | 2014-01-03 23:24:51 | 33 |
+-----+----------+-------------+---------------------+--------------+--------+------------+------+---------------+--------------------+---------------------------------------------------------------------------------------------------------------------+-------+--------------+---------+---------------------+-----------+
I see them being set to error state .
mysql> select * from snapshots where id in (336,337);
+-----+----------------+------------+-----------+-----------+------------------+--------+------+------------------------------+--------------------------------------+-------------+------------------+------------+---------------------+---------+----------------+----------+------------+--------------+-----------------+---------+------
| id | data_center_id | account_id | domain_id | volume_id | disk_offering_id
| status | path | name | uuid
| apshot_type | type_description | size | created |
removed | backup_snap_id | swift_id | sechost_id | prev_snap_id |
hypervisor_type | version | s3_id
+-----+----------------+------------+-----------+-----------+------------------+--------+------+------------------------------+--------------------------------------+-------------+------------------+------------+---------------------+---------+----------------+----------+------------+--------------+-----------------+---------+------
| 336 | 1 | 2 | 1 | 33 | 12
| Error | NULL | hello_ROOT-33_20140103230937 |
043d2ca4-14c4-4e02-99cb-306c4dc1413f | 0 | MANUAL |
8589934592 | 2014-01-03 23:09:37 | NULL | NULL | NULL |
NULL | NULL | KVM | 2.2 | NULL
| 337 | 1 | 2 | 1 | 33 | 12
| Error | NULL | hello_ROOT-33_20140103232449 |
87677b80-70f5-4bf6-84a2-e8103a85d904 | 0 | MANUAL |
8589934592 | 2014-01-03 23:24:49 | NULL | NULL | NULL |
NULL | NULL | KVM | 2.2 | NULL
+-----+----------------+------------+-----------+-----------+------------------+--------+------+------------------------------+--------------------------------------+-------------+------------------+------------+---------------------+---------+----------------+----------+------------+--------------+-----------------+---------+------
2 rows in set (0.00 sec)
But I still see the snapshots being listed:
[root@Rack3Host14 ~]# qemu-img info
/mnt/c20ea198-e8ca-33c3-9f11-e361ec9b5532/a90e68e2-6b9c-4ddf-9435-746dffe030de
image:
/mnt/c20ea198-e8ca-33c3-9f11-e361ec9b5532/a90e68e2-6b9c-4ddf-9435-746dffe030de
file format: qcow2
virtual size: 8.0G (8589934592 bytes)
disk size: 35M
cluster_size: 65536
backing file:
/mnt/c20ea198-e8ca-33c3-9f11-e361ec9b5532/63dd21e6-7188-11e3-a5ee-4adf980f9414
Snapshot list:
ID TAG VM SIZE DATE VM CLOCK
1 273939cf-0515-41c4-8553-5990b2770e2b 0 2014-01-03 15:10:42
00:00:00.000
2 c505ff5b-c4eb-4ad2-914b-9144c1ea5da2 0 2014-01-03 15:25:54
00:00:00.000
[root@Rack3Host14 ~]#
Reopening this issue to fix the clean up on primary store snapshot copy.
> KVM-Secondary Store down-Even after secondary store is brought back up after
> being down for few hours,snapshot jobs do not get triggered with reason
> "there is other active snapshot tasks on the instance to which the volume is
> attached"
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-5446
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5446
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Management Server
> Affects Versions: 4.3.0
> Environment: Build from 4.3
> Reporter: Sangeetha Hariharan
> Assignee: Min Chen
> Priority: Blocker
> Fix For: 4.3.0
>
> Attachments: agentdown.rar, ssdown.rar
>
>
> KVM - Secondary Store down - Even after secondary store is brought back up
> after being down for few hours , snapshot jobs do not get triggered with
> reason "here is other active snapshot tasks on the instance to which the
> volume is attached, please try again later"
> Set up:
> Advanced Zone set up with 2 KVM (RHEL 6.3) hosts.
> Steps to reproduce the problem:
> 1. Deploy 5 Vms in each of the hosts with 10 GB ROOT volume size , so we
> start with 10 Vms.
> 2. Start concurrent snapshots for ROOT volumes of all the Vms.
> 3. Shutdown the Secondary storage server when the snapshots are in the
> progress.
> 4. Bring the Secondary storage server up after ~ 12 hours.
> When the secondary storage was down:
> The snapshot jobs that were in progress timed out after 6 hours.
> Even after this , I do not see the hourly snapshots being scheduled due to
> the following reason:
>
> 2013-12-10 13:07:49,285 WARN [c.c.s.s.SnapshotSchedulerImpl]
> (SnapshotPollTask:ctx-cf0775f7) Scheduling snapshot failed due to
> com.cloud.utils.exception.CloudRuntimeException: There is other active
> snapshot tasks on the instance to which the volume is attached, please try
> again later
> Even after the secondary storage was brought up , there is no hourly
> snapshots being scheduled due to the same reason -
> "com.cloud.utils.exception.CloudRuntimeException: There is other active
> snapshot tasks on the instance to which the volume is attached, please try
> again later"
> mysql> select * FROM snapshots;
> +----+----------------+------------+-----------+-----------+------------------+------------------+------+-------------------------------------------------+--------------------------------------+---------------+------------------+-------------+---------------------+---------+----------------+----------+------------+--------------+-----------------+---------+-------+
> | id | data_center_id | account_id | domain_id | volume_id | disk_offering_id
> | status | path | name |
> uuid | snapshot_type | type_description |
> size | created | removed | backup_snap_id | swift_id |
> sechost_id | prev_snap_id | hypervisor_type | version | s3_id |
> +----+----------------+------------+-----------+-----------+------------------+------------------+------+-------------------------------------------------+--------------------------------------+---------------+------------------+-------------+---------------------+---------+----------------+----------+------------+--------------+-----------------+---------+-------+
> | 1 | 1 | 6 | 1 | 45 | 18
> | BackedUp | NULL | TestVM-tiny-host-0ps-0-0_ROOT-45_20131209234410 |
> ee2c46b7-7623-439a-9a30-63eec1d95c56 | 3 | HOURLY |
> 21474836480 | 2013-12-09 23:44:10 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 2 | 1 | 6 | 1 | 43 | 18
> | BackedUp | NULL | TestVM-1_ROOT-43_20131209234410 |
> 62c01389-49df-475b-b225-617d3903d25a | 3 | HOURLY |
> 21474836480 | 2013-12-09 23:44:10 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 3 | 1 | 6 | 1 | 54 | 19
> | BackedUp | NULL | TestVM-tiny-host-1ps-0-4_ROOT-54_20131209234410 |
> 8c7621f0-5d94-4f22-b3f8-7954c810e1be | 3 | HOURLY |
> 21474836480 | 2013-12-09 23:44:10 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 4 | 1 | 6 | 1 | 53 | 19
> | BackedUp | NULL | TestVM-tiny-host-1ps-0-3_ROOT-53_20131209234410 |
> 30c72de6-2a3e-4bf2-88da-b2a3b6ae9c66 | 3 | HOURLY |
> 21474836480 | 2013-12-09 23:44:10 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 5 | 1 | 6 | 1 | 52 | 19
> | BackedUp | NULL | TestVM-tiny-host-1ps-0-2_ROOT-52_20131209234411 |
> e8e9f99a-5a8b-4cfd-b743-327925955980 | 3 | HOURLY |
> 21474836480 | 2013-12-09 23:44:11 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 6 | 1 | 6 | 1 | 51 | 19
> | BackedUp | NULL | TestVM-tiny-host-1ps-0-1_ROOT-51_20131209234411 |
> 07e4edc8-b289-4f94-8696-ccf9db60e3ab | 3 | HOURLY |
> 21474836480 | 2013-12-09 23:44:11 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 7 | 1 | 6 | 1 | 50 | 19
> | BackedUp | NULL | TestVM-tiny-host-1ps-0-0_ROOT-50_20131209234411 |
> b285aa1a-6650-4b90-9c2c-e8bf0c507bfb | 3 | HOURLY |
> 21474836480 | 2013-12-09 23:44:11 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 8 | 1 | 6 | 1 | 49 | 18
> | BackedUp | NULL | TestVM-tiny-host-0ps-0-4_ROOT-49_20131209234412 |
> 31e37c8d-aab2-4ee6-9c43-c23501b08b02 | 3 | HOURLY |
> 21474836480 | 2013-12-09 23:44:12 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 9 | 1 | 6 | 1 | 48 | 18
> | BackedUp | NULL | TestVM-tiny-host-0ps-0-3_ROOT-48_20131209234412 |
> 60b6023e-138f-42c2-9867-f559e1d95352 | 3 | HOURLY |
> 21474836480 | 2013-12-09 23:44:12 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 10 | 1 | 6 | 1 | 47 | 18
> | CreatedOnPrimary | NULL | TestVM-tiny-host-0ps-0-2_ROOT-47_20131209234412 |
> fa37ebe5-f36c-4f19-b99f-249ab95c4e61 | 3 | HOURLY |
> 21474836480 | 2013-12-09 23:44:12 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 11 | 1 | 6 | 1 | 46 | 18
> | BackedUp | NULL | TestVM-tiny-host-0ps-0-1_ROOT-46_20131209234412 |
> bd571765-b28e-433e-b480-ccdc3bc0450b | 3 | HOURLY |
> 21474836480 | 2013-12-09 23:44:12 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 12 | 1 | 6 | 1 | 45 | 18
> | CreatedOnPrimary | NULL | TestVM-tiny-host-0ps-0-0_ROOT-45_20131210004410 |
> 4c6f50c2-661e-4bad-9f07-357b2ca4e7fc | 3 | HOURLY |
> 21474836480 | 2013-12-10 00:44:10 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 13 | 1 | 6 | 1 | 43 | 18
> | CreatedOnPrimary | NULL | TestVM-1_ROOT-43_20131210004410 |
> bd3744ae-182f-4a0b-9685-5bef0c747365 | 3 | HOURLY |
> 21474836480 | 2013-12-10 00:44:10 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 14 | 1 | 6 | 1 | 54 | 19
> | CreatedOnPrimary | NULL | TestVM-tiny-host-1ps-0-4_ROOT-54_20131210004410 |
> 14f53a2f-bb4b-40e5-aa90-e7b4be4a1076 | 3 | HOURLY |
> 21474836480 | 2013-12-10 00:44:10 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 15 | 1 | 6 | 1 | 53 | 19
> | CreatedOnPrimary | NULL | TestVM-tiny-host-1ps-0-3_ROOT-53_20131210004410 |
> 88dd82f5-ade5-4dd5-a147-e0a35a15be85 | 3 | HOURLY |
> 21474836480 | 2013-12-10 00:44:10 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 16 | 1 | 6 | 1 | 52 | 19
> | CreatedOnPrimary | NULL | TestVM-tiny-host-1ps-0-2_ROOT-52_20131210004410 |
> 1830af7d-97e5-488c-bc37-39c11961e509 | 3 | HOURLY |
> 21474836480 | 2013-12-10 00:44:10 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 17 | 1 | 6 | 1 | 51 | 19
> | CreatedOnPrimary | NULL | TestVM-tiny-host-1ps-0-1_ROOT-51_20131210004410 |
> 3d452970-f042-4af6-a8cc-3003482da9a8 | 3 | HOURLY |
> 21474836480 | 2013-12-10 00:44:10 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 18 | 1 | 6 | 1 | 50 | 19
> | CreatedOnPrimary | NULL | TestVM-tiny-host-1ps-0-0_ROOT-50_20131210004410 |
> 0cb1b176-bcfe-4bc2-9b5e-07ec8e9e97f5 | 3 | HOURLY |
> 21474836480 | 2013-12-10 00:44:10 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 19 | 1 | 6 | 1 | 49 | 18
> | CreatedOnPrimary | NULL | TestVM-tiny-host-0ps-0-4_ROOT-49_20131210004411 |
> 87337d8b-e698-47b1-961a-32479d44a6e5 | 3 | HOURLY |
> 21474836480 | 2013-12-10 00:44:11 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 20 | 1 | 6 | 1 | 48 | 18
> | CreatedOnPrimary | NULL | TestVM-tiny-host-0ps-0-3_ROOT-48_20131210004411 |
> c7b321bd-31b4-4c9c-be8f-c9231cb08530 | 3 | HOURLY |
> 21474836480 | 2013-12-10 00:44:11 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 21 | 1 | 6 | 1 | 47 | 18
> | CreatedOnPrimary | NULL | TestVM-tiny-host-0ps-0-2_ROOT-47_20131210004411 |
> ea1a71b0-9b52-42e5-a252-7a3e8938bc0f | 3 | HOURLY |
> 21474836480 | 2013-12-10 00:44:11 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 22 | 1 | 6 | 1 | 46 | 18
> | CreatedOnPrimary | NULL | TestVM-tiny-host-0ps-0-1_ROOT-46_20131210004411 |
> f1fbc899-5f55-472c-b798-639350b2facd | 3 | HOURLY |
> 21474836480 | 2013-12-10 00:44:11 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 33 | 1 | 6 | 1 | 45 | 18
> | CreatedOnPrimary | NULL | TestVM-tiny-host-0ps-0-0_ROOT-45_20131210022749 |
> 396c80a8-299c-42b2-8c3d-32da5af9536c | 3 | HOURLY |
> 21474836480 | 2013-12-10 02:27:49 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 34 | 1 | 6 | 1 | 43 | 18
> | CreatedOnPrimary | NULL | TestVM-1_ROOT-43_20131210022749 |
> 9444a355-08a6-4c7d-98f2-74729a0b28b5 | 3 | HOURLY |
> 21474836480 | 2013-12-10 02:27:49 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 35 | 1 | 6 | 1 | 54 | 19
> | CreatedOnPrimary | NULL | TestVM-tiny-host-1ps-0-4_ROOT-54_20131210022749 |
> 09cbfc03-7f9f-470f-bc34-e2bbad72b9ba | 3 | HOURLY |
> 21474836480 | 2013-12-10 02:27:49 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 36 | 1 | 6 | 1 | 53 | 19
> | CreatedOnPrimary | NULL | TestVM-tiny-host-1ps-0-3_ROOT-53_20131210022749 |
> 2e7259f0-82d2-4910-9047-617d54bda035 | 3 | HOURLY |
> 21474836480 | 2013-12-10 02:27:49 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 37 | 1 | 6 | 1 | 52 | 19
> | CreatedOnPrimary | NULL | TestVM-tiny-host-1ps-0-2_ROOT-52_20131210022749 |
> 420230ed-a037-4ed3-a019-031d24fd0bfb | 3 | HOURLY |
> 21474836480 | 2013-12-10 02:27:49 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 38 | 1 | 6 | 1 | 51 | 19
> | CreatedOnPrimary | NULL | TestVM-tiny-host-1ps-0-1_ROOT-51_20131210022750 |
> 6bf87e88-3a14-467f-ac5b-ab620fd03206 | 3 | HOURLY |
> 21474836480 | 2013-12-10 02:27:50 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 39 | 1 | 6 | 1 | 50 | 19
> | CreatedOnPrimary | NULL | TestVM-tiny-host-1ps-0-0_ROOT-50_20131210022750 |
> 93021350-9df1-4234-9df2-869d9e9a912f | 3 | HOURLY |
> 21474836480 | 2013-12-10 02:27:50 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 40 | 1 | 6 | 1 | 49 | 18
> | CreatedOnPrimary | NULL | TestVM-tiny-host-0ps-0-4_ROOT-49_20131210022750 |
> 4699ebdf-3093-4f2f-86b7-ae5c9195cc6a | 3 | HOURLY |
> 21474836480 | 2013-12-10 02:27:50 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 41 | 1 | 6 | 1 | 48 | 18
> | CreatedOnPrimary | NULL | TestVM-tiny-host-0ps-0-3_ROOT-48_20131210022750 |
> 8d663979-31a2-48ad-884b-f1ef85a1f4d5 | 3 | HOURLY |
> 21474836480 | 2013-12-10 02:27:50 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 42 | 1 | 6 | 1 | 46 | 18
> | CreatedOnPrimary | NULL | TestVM-tiny-host-0ps-0-1_ROOT-46_20131210022751 |
> f80c2994-b612-4234-add4-27d323cf3dd3 | 3 | HOURLY |
> 21474836480 | 2013-12-10 02:27:51 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> +----+----------------+------------+-----------+-----------+------------------+------------------+------+-------------------------------------------------+--------------------------------------+---------------+------------------+-------------+---------------------+---------+----------------+----------+------------+--------------+-----------------+---------+-------+
> 32 rows in set (0.00 sec)
> mysql>
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
