weizhouapache commented on issue #9501: URL: https://github.com/apache/cloudstack/issues/9501#issuecomment-5475616210
**Context:** This assumes CloudStack 4.19.0.1, XCP-ng/XenServer 8.2, `snapshot.delta.max=7`, with a daily snapshot policy and Keep=3. **Diagnosis:** Known bug. The retention/GC path is in `DefaultSnapshotStrategy.deleteSnapshotChain()` / `deleteSnapshotInfo()` (`engine/storage/snapshot/src/main/java/org/apache/cloudstack/storage/snapshot/DefaultSnapshotStrategy.java`). When the retention sweep tries to delete a snapshot that is still a live "parent" in the VHD chain (i.e. it has a `child`), `deleteSnapshotChain()` correctly skips the physical delete to avoid corrupting the child's backing file — but its caller unconditionally fires `Snapshot.Event.OperationSucceeded` anyway, flipping the top‑level `Snapshot` DB record to `Destroyed` while the underlying `SnapshotDataStoreVO` row (and the VHD file itself) is left `display=false` but still `Ready`. Nothing in 4.19.x later revisits that row to retry cleanup once the child is retired, so it leaks indefinitely. This matches your report: some intermediate snapshots (no live child at delete time) get cleaned up fine, while others (still a live parent at delete time ) are marked "gone" in the UI/DB but never actually removed from storage. A `snapshot.delta.max=7` chain with Keep=3 makes this trigger often, since the oldest snapshot chosen for deletion frequently still has a live child somewhere in the still-growing 7-deep chain. **Resolution:** - Upgrade to at least **4.19.2.0**, which includes [PR #9447](https://github.com/apache/cloudstack/pull/9447) (fixes [#9446](https://github.com/apache/cloudstack/issues/9446)). Note this fixes the more severe half of this code's history — parents being erroneously *erased* while still live — but does **not** fully resolve the orphaned-`Ready`-row leak described here. - The fix that directly addresses this exact symptom is [PR #12597](https://github.com/apache/cloudstack/pull/12597) (fixes [#12524](https://github.com/apache/cloudstack/issues/12524)), which replaces the binary Ready/Destroyed model with a proper `Hidden` state for "retired but still needed as chain backing storage." This landed in **4.22.1** — there is no backport to 4.19/4.20 at this time. - Until you can upgrade to 4.22.1+, treat the leaked files as a known limitation. If you need to reclaim space, audit `snapshot_store_ref` for rows with `display=0` and `state=Ready` whose parent `snapshot` row is `Destroyed`, confirm no active volume/template depends on that VHD, and clean up manually — do this carefully and only after confirming the chain is truly orphaned, since incorrect manual deletion on a live chain can cause data loss (the exact failure mode #9447 fixed). **Sources:** [#9446](https://github.com/apache/cloudstack/issues/9446) / [#9447](https://github.com/apache/cloudstack/pull/9447) (merged 4.19.2.0, commit `9033ab709e6`); [#12524](https://github.com/apache/cloudstack/issues/12524) / [#12597](https://github.com/apache/cloudstack/pull/12597) (merged 4.22.1); code read at tag `4.19.0.1` in `DefaultSnapshotStrategy.java`. **Next steps:** If you can reproduce reliably after uprgade, attaching the `snapshot_store_ref` state for one orphaned chain (snapshot IDs, `state`, `display`, `parent_snapshot_id`) here would help confirm this is the same mechanism rather than a distinct regression. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
