rangareddy commented on issue #11122: URL: https://github.com/apache/hudi/issues/11122#issuecomment-4855132239
This is a clean-planning OOM: the message is `Requested array size exceeds VM limit` (the ~2GB single-array cap), not heap — so more driver memory won't help. The trigger is a non-incremental (full) clean that enumerates every partition/file group and collects it on the driver at `CleanPlanActionExecutor`. That happens when the incremental-clean reference commit gets archived out of the active timeline (your `retained=449` / `keep.min=450` leaves only a one-commit margin, so a single missed clean forces a full clean next time) — which is also why dropping `keep.max.commits` + manual archive helps for a while then recurs. Check the driver log for `Total partitions to clean : N`; if N is your whole partition count, that's the full-clean path. Fixes: (1) enable the metadata table (`hoodie.metadata.enable=true`) so the clean plan's listing is served from MDT instead of a giant driver-side array; (2) widen the retention gap (e.g. retained≪keep.min≪keep.max) and ensure clean run s and completes every commit so its reference is never archived; (3) reduce file-group cardinality via small-file handling/clustering; and (4) upgrade off 0.12.1, where clean planning/MDT listing were improved (this is the #8199/#7800 lineage). Increasing memory won't resolve the array-size cap. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
