Stephen0421 opened a new pull request, #9609: URL: https://github.com/apache/paimon/pull/9609
### Purpose This is the first PR split from #9207. PK managed BLOB packs (`.managed.blob`) are shared payloads, not first-class data files. Compaction reuses pack bytes without copying them, so snapshot expire only deletes the data file and its `.blobref` extra file. Unreferenced packs previously accumulated forever. This change reclaims those packs through `LocalManagedBlobOrphanFilesClean`: - Collect reachability from live `ADD` data files' `.blobref` sidecars. - Join candidates by full pack identity (`storageRootId + relativePath`), not basename, so same-named packs in different directories are not mixed. Qualified and unqualified paths use the same URI path so `hdfs:///warehouse/...` still matches `hdfs://nn:8020/warehouse/...`. - Collect used packs twice and abort the run (delete nothing) if the snapshot topology or used-pack set changed between those collections. That shrinks the compaction-reuse race; it is not a commit lease. - Delete unused `.managed.blob` files older than `older_than` (1 day by default). - If a listed sidecar on a **still-existing** data file is missing, corrupt, or unsupported, skip **all** pack deletes for that run. An empty valid sidecar is not missing: it means the file references no pack. - Unmerged `ADD` entries whose data files are already gone after expire are ignored, so they do not abort pack GC. `remove_orphan_files` never deletes `.managed.blob` packs. This is a conservative, best-effort first version. There is still no commit lease. Keep a non-zero `older_than`. Do not treat this as strictly proven concurrent-safe delete semantics. Flink and Spark `remove_orphan_blobs` procedures are intentionally out of this PR and will follow in stacked PRs. ### Tests - `ManagedBlobReachabilityCollectorTest`: empty extras / empty sidecar, referenced packs, missing sidecar (live vs already gone), corrupt / unsupported version, unsafe merge. - `ManagedBlobOrphanFilesCleanTest`: delete unreferenced pack, keep referenced pack, join by full pack path, empty sidecar does not block others, missing / corrupt / unsupported sidecar skips all packs, unreferenced pack after update+expire, abort when used set changes between the two collections, compaction interleaving after the final mark, and database executor cancellation / bounded termination. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
