Stephen0421 commented on PR #10014: URL: https://github.com/apache/paimon/pull/10014#issuecomment-5756931828
> Requirement fit: SUPPORTED (triage: GO)\nImplementation: FINDINGS\n\nThe Flink action and procedure have clear end-to-end operational value, but I found one scalability blocker in the distributed path.\n\n[P1] Avoid copying the complete used-pack set into every cleanup task\n\nFlinkManagedBlobOrphanFilesClean.java freezes all live pack identities into a client HashSet, then deleteUnused copies that set and captures it in the candidate filter. Because that filter runs at the requested parallelism, each subtask receives the complete N-entry set, making worker heap usage O(P×N) for N live packs and P cleanup tasks. Live packs scale with retained snapshots and files, so increasing parallelism can make the distributed cleaner OOM on exactly the large tables it is intended to handle.\n\nPlease turn the frozen identities into a lineage-detached bounded source and use a keyed bounded anti-join with candidates, as FlinkOrphanFilesClean already does, so each task retains only its key par tition while preserving the no-recomputed-mark safety property. The current 256-orphan scale case has a small live set and does not exercise this amplification. Thanks for the catch. The deletion filter captured the full frozen HashSet, so each cleanup subtask held all N identities (O(P×N)). Deletion now re-sources the frozen names with `fromCollection` (lineage-detached, parallelism 1) and keyed-joins them with candidates, same pattern as `FlinkOrphanFilesClean`. Each join task only keeps its key partition. The freeze `executeAndCollect` is unchanged, so the deletion job still cannot recompute sidecars. An empty used set skips `fromCollection` (Flink rejects an empty collection) and treats every candidate as unused. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
