sureshanaparti opened a new pull request, #12347: URL: https://github.com/apache/cloudstack/pull/12347
### Description This PR addresses cleanup of the snapshots in datastores for Error-ed snapshots, and has some code improvements. The removed date is set for the snapshot records after backup failed [1], and the storage garbage collector ignores cleaning up the files as getSnapshot [2] always returns null, leaving the orphaned snapshot files in the primary storage. [1] https://github.com/apache/cloudstack/blob/4.20/engine/storage/snapshot/src/main/java/org/apache/cloudstack/storage/snapshot/SnapshotServiceImpl.java#L408 https://github.com/apache/cloudstack/blob/4.20/engine/storage/snapshot/src/main/java/org/apache/cloudstack/storage/snapshot/SnapshotServiceImpl.java#L719 [2] https://github.com/apache/cloudstack/blob/4.20/server/src/main/java/com/cloud/storage/StorageManagerImpl.java#L1889 <!--- Describe your changes in DETAIL - And how has behaviour functionally changed. --> <!-- For new features, provide link to FS, dev ML discussion etc. --> <!-- In case of bug fix, the expected and actual behaviours, steps to reproduce. --> <!-- When "Fixes: #<id>" is specified, the issue/PR will automatically be closed when this PR gets merged --> <!-- For addressing multiple issues/PRs, use multiple "Fixes: #<id>" --> <!-- Fixes: # --> <!--- ******************************************************************************* --> <!--- NOTE: AUTOMATION USES THE DESCRIPTIONS TO SET LABELS AND PRODUCE DOCUMENTATION. --> <!--- PLEASE PUT AN 'X' in only **ONE** box --> <!--- ******************************************************************************* --> ### Types of changes - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] New feature (non-breaking change which adds functionality) - [ ] Bug fix (non-breaking change which fixes an issue) - [x] Enhancement (improves an existing feature and functionality) - [ ] Cleanup (Code refactoring and cleanup, that may add test cases) - [ ] Build/CI - [ ] Test (unit or integration test code) ### Feature/Enhancement Scale or Bug Severity #### Feature/Enhancement Scale - [ ] Major - [x] Minor #### Bug Severity - [ ] BLOCKER - [ ] Critical - [ ] Major - [ ] Minor - [ ] Trivial ### Screenshots (if appropriate): ### How Has This Been Tested? Failed snapshots after created in primary (before copy to secondary storage) by adding the below code, and verified the cleanup before and after changes. ``` diff --git a/engine/storage/datamotion/src/main/java/org/apache/cloudstack/storage/motion/AncientDataMotionStrategy.java b/engine/storage/datamotion/src/main/java/org/apache/cloudstack/storage/motion/AncientDataMotionStrategy.java index 0b0065361d0..66ad50270a8 100644 --- a/engine/storage/datamotion/src/main/java/org/apache/cloudstack/storage/motion/AncientDataMotionStrategy.java +++ b/engine/storage/datamotion/src/main/java/org/apache/cloudstack/storage/motion/AncientDataMotionStrategy.java @@ -606,6 +606,10 @@ public class AncientDataMotionStrategy implements DataMotionStrategy { Answer answer = null; try { + if (srcData.getType() == DataObjectType.SNAPSHOT && destData.getType() == DataObjectType.SNAPSHOT) { + throw new CloudRuntimeException("*** FAIL COPY SNAPSHOT ***"); + } + if (needCacheStorage(srcData, destData)) { Scope selectedScope = pickCacheScopeForCopy(srcData, destData); cacheData = cacheMgr.getCacheObject(srcData, selectedScope); ``` **BEFORE CHANGES:** Logs: ``` [root@mgmt1 ~]# grep "copy snasphot failed" /var/log/cloudstack/management/management-server.log 2025-12-29 07:55:33,741 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] (Work-Job-Executor-1:[ctx-e1e63336, job-383/job-384, ctx-d5307dd2]) (logid:eee170fc) copy snasphot failed: com.cloud.utils.exception.CloudRuntimeException: *** FAIL COPY SNAPSHOT *** 2025-12-29 07:56:40,862 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] (Work-Job-Executor-2:[ctx-23052f31, job-385/job-386, ctx-2ae57c30]) (logid:c8826374) copy snasphot failed: com.cloud.utils.exception.CloudRuntimeException: *** FAIL COPY SNAPSHOT *** 2025-12-29 08:48:46,174 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] (Work-Job-Executor-3:[ctx-62f08aef, job-387/job-388, ctx-8d527b2a]) (logid:9ffd7643) copy snasphot failed: com.cloud.utils.exception.CloudRuntimeException: *** FAIL COPY SNAPSHOT *** 2025-12-29 09:48:43,056 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] (Work-Job-Executor-4:[ctx-4148adbb, job-389/job-390, ctx-ed1dc303]) (logid:e2a3192f) copy snasphot failed: com.cloud.utils.exception.CloudRuntimeException: *** FAIL COPY SNAPSHOT *** 2025-12-29 10:48:41,956 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] (Work-Job-Executor-5:[ctx-844b61a8, job-391/job-392, ctx-7a894e61]) (logid:8ad0e711) copy snasphot failed: com.cloud.utils.exception.CloudRuntimeException: *** FAIL COPY SNAPSHOT *** 2025-12-29 11:35:25,124 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] (Work-Job-Executor-1:[ctx-5585d9a9, job-395/job-396, ctx-be6b1bff]) (logid:4d367e53) copy snasphot failed: com.cloud.utils.exception.CloudRuntimeException: *** FAIL COPY SNAPSHOT *** 2025-12-29 11:48:49,195 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] (Work-Job-Executor-2:[ctx-0ae9bdea, job-397/job-398, ctx-5d0cbc49]) (logid:ed5b9de8) copy snasphot failed: com.cloud.utils.exception.CloudRuntimeException: *** FAIL COPY SNAPSHOT *** 2025-12-29 12:48:47,689 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] (Work-Job-Executor-3:[ctx-deaf5254, job-399/job-400, ctx-c48773b2]) (logid:03ac9fda) copy snasphot failed: com.cloud.utils.exception.CloudRuntimeException: *** FAIL COPY SNAPSHOT *** 2025-12-29 12:48:58,623 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] (Work-Job-Executor-4:[ctx-be293b7e, job-401/job-402, ctx-6d3ce524]) (logid:fdc87173) copy snasphot failed: com.cloud.utils.exception.CloudRuntimeException: *** FAIL COPY SNAPSHOT *** ``` DB: ``` mysql> SELECT id, uuid, name, volume_id, status, path, snapshot_type, type_description, size, created, removed FROM cloud.snapshots WHERE status = 'Error'; +-----+--------------------------------------+--------------------------------+-----------+--------+------+---------------+------------------+------------+---------------------+---------------------+ | id | uuid | name | volume_id | status | path | snapshot_type | type_description | size | created | removed | +-----+--------------------------------------+--------------------------------+-----------+--------+------+---------------+------------------+------------+---------------------+---------------------+ | 127 | e7c5ec0e-a179-4ed5-a1bf-cd4cf1d03f5d | ROOT-3-Snap04_ERROR | 3 | Error | NULL | 0 | MANUAL | 8589934592 | 2025-12-29 07:55:21 | 2025-12-29 07:55:33 | | 128 | 350a90af-d0f4-4474-b10b-befac336cbd3 | ROOT-3-Snap05_ERROR | 3 | Error | NULL | 0 | MANUAL | 8589934592 | 2025-12-29 07:56:30 | 2025-12-29 07:56:40 | | 129 | 8a07c40f-d09d-4ced-b959-382ac6cc93ca | testvm01_ROOT-3_20251229084828 | 3 | Error | NULL | 3 | HOURLY | 8589934592 | 2025-12-29 08:48:28 | 2025-12-29 08:48:46 | | 130 | d84186e6-3e26-432b-95d0-67f6883cea6a | testvm01_ROOT-3_20251229094828 | 3 | Error | NULL | 3 | HOURLY | 8589934592 | 2025-12-29 09:48:28 | 2025-12-29 09:48:43 | | 131 | b586c299-d46f-4599-ae60-6852c3e913b1 | testvm01_ROOT-3_20251229104828 | 3 | Error | NULL | 3 | HOURLY | 8589934592 | 2025-12-29 10:48:28 | 2025-12-29 10:48:41 | | 132 | e068e33d-4139-4a21-bdd3-a1e1a7423c57 | ROOT-3-Snap06_ERROR | 3 | Error | NULL | 0 | MANUAL | 8589934592 | 2025-12-29 11:35:10 | 2025-12-29 11:35:25 | | 133 | 921ac9cb-ab3e-4d29-9b67-d31a0bcb313a | testvm01_ROOT-3_20251229114834 | 3 | Error | NULL | 3 | HOURLY | 8589934592 | 2025-12-29 11:48:34 | 2025-12-29 11:48:49 | | 134 | fc462ef5-48b5-46d8-8d2c-301db0653d5e | testvm01_ROOT-3_20251229124834 | 3 | Error | NULL | 3 | HOURLY | 8589934592 | 2025-12-29 12:48:34 | 2025-12-29 12:48:47 | | 135 | 3e575604-2d32-4a3e-802e-e0209267c508 | testvm01_ROOT-3_20251229124834 | 3 | Error | NULL | 4 | DAILY | 8589934592 | 2025-12-29 12:48:34 | 2025-12-29 12:48:58 | +-----+--------------------------------------+--------------------------------+-----------+--------+------+---------------+------------------+------------+---------------------+---------------------+ 9 rows in set (0.00 sec) mysql> SELECT id, snapshot_id, store_id, store_role, size, physical_size, install_path, local_path, state, volume_id, download_state, download_pct, error_str, created, last_updated FROM cloud.snapshot_store_ref WHERE snapshot_id IN (SELECT id FROM cloud.snapshots WHERE status = 'Error'); +-----+-------------+----------+------------+------------+---------------+------------------------------------------------------------------------------------------+------------+-----------+-----------+----------------+--------------+-----------+---------------------+--------------+ | id | snapshot_id | store_id | store_role | size | physical_size | install_path | local_path | state | volume_id | download_state | download_pct | error_str | created | last_updated | +-----+-------------+----------+------------+------------+---------------+------------------------------------------------------------------------------------------+------------+-----------+-----------+----------------+--------------+-----------+---------------------+--------------+ | 253 | 127 | 1 | Primary | 8589934592 | 8589934592 | /mnt/4d215758-18a9-335e-8c9d-ecb4d19c9d94/snapshots/ac022550-1d39-4e01-8b10-d18634abbc71 | NULL | Destroyed | 3 | NULL | 0 | NULL | 2025-12-29 07:55:22 | NULL | | 255 | 128 | 1 | Primary | 8589934592 | 8589934592 | /mnt/4d215758-18a9-335e-8c9d-ecb4d19c9d94/snapshots/b5b8e306-c015-4c21-b9da-7abf0536797f | NULL | Destroyed | 3 | NULL | 0 | NULL | 2025-12-29 07:56:30 | NULL | | 257 | 129 | 1 | Primary | 8589934592 | 8589934592 | /mnt/4d215758-18a9-335e-8c9d-ecb4d19c9d94/snapshots/649c946a-7520-4838-945e-e229e4fd187b | NULL | Destroyed | 3 | NULL | 0 | NULL | 2025-12-29 08:48:30 | NULL | | 259 | 130 | 1 | Primary | 8589934592 | 8589934592 | /mnt/4d215758-18a9-335e-8c9d-ecb4d19c9d94/snapshots/ce440897-bf02-450e-914e-3ed4f4b5c05d | NULL | Destroyed | 3 | NULL | 0 | NULL | 2025-12-29 09:48:30 | NULL | | 261 | 131 | 1 | Primary | 8589934592 | 8589934592 | /mnt/4d215758-18a9-335e-8c9d-ecb4d19c9d94/snapshots/81f9ee83-9de3-4a75-9082-9e27fe844151 | NULL | Destroyed | 3 | NULL | 0 | NULL | 2025-12-29 10:48:30 | NULL | | 263 | 132 | 1 | Primary | 8589934592 | 8589934592 | /mnt/4d215758-18a9-335e-8c9d-ecb4d19c9d94/snapshots/8638a61a-ca2a-46ca-a166-8ae16275ec6a | NULL | Destroyed | 3 | NULL | 0 | NULL | 2025-12-29 11:35:11 | NULL | | 265 | 133 | 1 | Primary | 8589934592 | 8589934592 | /mnt/4d215758-18a9-335e-8c9d-ecb4d19c9d94/snapshots/705ddb09-0f23-411a-a529-50545440fe28 | NULL | Destroyed | 3 | NULL | 0 | NULL | 2025-12-29 11:48:35 | NULL | | 267 | 134 | 1 | Primary | 8589934592 | 8589934592 | /mnt/4d215758-18a9-335e-8c9d-ecb4d19c9d94/snapshots/81ff1ba0-08ab-4ea3-9206-fde27b5ce53d | NULL | Destroyed | 3 | NULL | 0 | NULL | 2025-12-29 12:48:35 | NULL | | 269 | 135 | 1 | Primary | 8589934592 | 8589934592 | /mnt/4d215758-18a9-335e-8c9d-ecb4d19c9d94/snapshots/4e402bf0-1e47-4e23-bb5e-c20da9c94135 | NULL | Destroyed | 3 | NULL | 0 | NULL | 2025-12-29 12:48:47 | NULL | +-----+-------------+----------+------------+------------+---------------+------------------------------------------------------------------------------------------+------------+-----------+-----------+----------------+--------------+-----------+---------------------+--------------+ 9 rows in set (0.00 sec) ``` Files in Primary Storage: ``` [root@kvm1 ~]# ls -lrt /mnt/4d215758-18a9-335e-8c9d-ecb4d19c9d94/snapshots/ total 14611728 -rw-r--r--. 1 root root 1659240448 Dec 29 07:55 ac022550-1d39-4e01-8b10-d18634abbc71 -rw-r--r--. 1 root root 1659240448 Dec 29 07:56 b5b8e306-c015-4c21-b9da-7abf0536797f -rw-r--r--. 1 root root 1659240448 Dec 29 08:48 649c946a-7520-4838-945e-e229e4fd187b -rw-r--r--. 1 root root 1659240448 Dec 29 09:48 ce440897-bf02-450e-914e-3ed4f4b5c05d -rw-r--r--. 1 root root 1659240448 Dec 29 10:48 81f9ee83-9de3-4a75-9082-9e27fe844151 -rw-r--r--. 1 root root 1659240448 Dec 29 11:35 8638a61a-ca2a-46ca-a166-8ae16275ec6a -rw-r--r--. 1 root root 1659240448 Dec 29 11:48 705ddb09-0f23-411a-a529-50545440fe28 -rw-r--r--. 1 root root 1659240448 Dec 29 12:48 81ff1ba0-08ab-4ea3-9206-fde27b5ce53d -rw-r--r--. 1 root root 1659240448 Dec 29 12:48 4e402bf0-1e47-4e23-bb5e-c20da9c94135 ``` **AFTER CHANGES:** Logs: ``` [root@mgmt1 ~]# grep "state found on" /var/log/cloudstack/management/management-server.log 2025-12-29 13:40:28,262 DEBUG [c.c.s.StorageManagerImpl] (StorageManager-Scavenger-1:[ctx-40654b77]) (logid:09807cbb) Snapshot [e7c5ec0e-a179-4ed5-a1bf-cd4cf1d03f5d] in Destroyed state found on primary data store [4d215758-18a9-335e-8c9d-ecb4d19c9d94]; therefore, it will be destroyed. 2025-12-29 13:40:28,467 DEBUG [c.c.s.StorageManagerImpl] (StorageManager-Scavenger-1:[ctx-40654b77]) (logid:09807cbb) Snapshot [350a90af-d0f4-4474-b10b-befac336cbd3] in Destroyed state found on primary data store [4d215758-18a9-335e-8c9d-ecb4d19c9d94]; therefore, it will be destroyed. 2025-12-29 13:40:28,607 DEBUG [c.c.s.StorageManagerImpl] (StorageManager-Scavenger-1:[ctx-40654b77]) (logid:09807cbb) Snapshot [8a07c40f-d09d-4ced-b959-382ac6cc93ca] in Destroyed state found on primary data store [4d215758-18a9-335e-8c9d-ecb4d19c9d94]; therefore, it will be destroyed. 2025-12-29 13:40:28,752 DEBUG [c.c.s.StorageManagerImpl] (StorageManager-Scavenger-1:[ctx-40654b77]) (logid:09807cbb) Snapshot [d84186e6-3e26-432b-95d0-67f6883cea6a] in Destroyed state found on primary data store [4d215758-18a9-335e-8c9d-ecb4d19c9d94]; therefore, it will be destroyed. 2025-12-29 13:40:28,942 DEBUG [c.c.s.StorageManagerImpl] (StorageManager-Scavenger-1:[ctx-40654b77]) (logid:09807cbb) Snapshot [b586c299-d46f-4599-ae60-6852c3e913b1] in Destroyed state found on primary data store [4d215758-18a9-335e-8c9d-ecb4d19c9d94]; therefore, it will be destroyed. 2025-12-29 13:40:29,094 DEBUG [c.c.s.StorageManagerImpl] (StorageManager-Scavenger-1:[ctx-40654b77]) (logid:09807cbb) Snapshot [e068e33d-4139-4a21-bdd3-a1e1a7423c57] in Destroyed state found on primary data store [4d215758-18a9-335e-8c9d-ecb4d19c9d94]; therefore, it will be destroyed. 2025-12-29 13:40:29,271 DEBUG [c.c.s.StorageManagerImpl] (StorageManager-Scavenger-1:[ctx-40654b77]) (logid:09807cbb) Snapshot [921ac9cb-ab3e-4d29-9b67-d31a0bcb313a] in Destroyed state found on primary data store [4d215758-18a9-335e-8c9d-ecb4d19c9d94]; therefore, it will be destroyed. 2025-12-29 13:40:29,482 DEBUG [c.c.s.StorageManagerImpl] (StorageManager-Scavenger-1:[ctx-40654b77]) (logid:09807cbb) Snapshot [fc462ef5-48b5-46d8-8d2c-301db0653d5e] in Destroyed state found on primary data store [4d215758-18a9-335e-8c9d-ecb4d19c9d94]; therefore, it will be destroyed. 2025-12-29 13:40:29,760 DEBUG [c.c.s.StorageManagerImpl] (StorageManager-Scavenger-1:[ctx-40654b77]) (logid:09807cbb) Snapshot [3e575604-2d32-4a3e-802e-e0209267c508] in Destroyed state found on primary data store [4d215758-18a9-335e-8c9d-ecb4d19c9d94]; therefore, it will be destroyed. ``` DB: ``` mysql> SELECT id, uuid, name, volume_id, status, path, snapshot_type, type_description, size, created, removed FROM cloud.snapshots WHERE status = 'Error'; Empty set (0.00 sec) mysql> SELECT id, snapshot_id, store_id, store_role, size, physical_size, install_path, local_path, state, volume_id, download_state, download_pct, error_str, created, last_updated FROM cloud.snapshot_store_ref WHERE snapshot_id IN (SELECT id FROM cloud.snapshots WHERE status = 'Error'); Empty set (0.00 sec) ``` Files in Primary Storage: ``` [root@kvm1 ~]# ls -lrt /mnt/4d215758-18a9-335e-8c9d-ecb4d19c9d94/snapshots/ total 0 ``` <!-- Please describe in detail how you tested your changes. --> <!-- Include details of your testing environment, and the tests you ran to --> #### How did you try to break this feature and the system with this change? <!-- see how your change affects other areas of the code, etc. --> <!-- Please read the [CONTRIBUTING](https://github.com/apache/cloudstack/blob/main/CONTRIBUTING.md) document --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
