Stephen0421 opened a new pull request, #9207: URL: https://github.com/apache/paimon/pull/9207
### Purpose PK managed BLOB packs (`.managed.blob`) are shared payloads, not first-class data files. Compaction reuses pack bytes without copying them, so snapshot expire only deletes the data file and its `.blobref` extra file. Unreferenced packs previously accumulated forever. This change reclaims those packs through the existing `remove_orphan_files` path (local / Flink / Spark): - Collect reachability from live `ADD` data files' `.blobref` sidecars. - Delete unused `.managed.blob` files older than `older_than` (1 day by default). - If a listed sidecar on a **still-existing** data file is missing, corrupt, or unsupported, skip **all** pack deletes for that run. An empty valid sidecar is not missing: it means the file references no pack. - Unmerged `ADD` entries whose data files are already gone after expire are ignored, so they do not abort pack GC. This is a conservative, best-effort first version. There is no commit lease. Compaction can reuse a pack without refreshing mtime, so `older_than` is not a concurrent delete fence for long-lived reused packs. Keep a non-zero `older_than`. Do not treat this as strictly proven concurrent-safe delete semantics. Out of scope: expire-time incremental pack delete, `FileStoreCommitImpl.abort` pack cleanup, and Append / Data Evolution `.blob` files (already handled as ordinary data files). ### Tests - `ManagedBlobReachabilityCollectorTest`: empty extras / empty sidecar, referenced packs, missing sidecar (live vs already gone), corrupt / unsupported version, unsafe merge. - `ManagedBlobOrphanFilesCleanTest`: delete unreferenced pack, keep referenced pack, empty sidecar does not block others, missing / corrupt / unsupported sidecar skips all packs, unreferenced pack after update+expire, and the deterministic compact+expire interleaving that shows a stale snapshot list neither skips nor retains a reused live pack. - `LocalOrphanFilesCleanTest`: previously kept all managed packs; now unused packs are eligible. - Flink `RemoveOrphanFilesActionITCaseBase`: local and distributed — delete unreferenced pack; missing sidecar skips pack GC but still deletes other orphans. - Spark `ManagedBlobOrphanFilesProcedureTest`: same two cases × local / distributed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
