ehurheap opened a new pull request, #19377:
URL: https://github.com/apache/hudi/pull/19377
### opt-in batched clean plans for very-large-partition tables
Adds hoodie.clean.plan.partitions.batch.size (default -1, disabled).
When set to a positive value, a single logical clean is paginated into
multiple internally-batched clean instants, each covering at most N
partitions. This avoids serializing one giant HoodieCleanerPlan Avro
array and hitting the JVM Integer.MAX_VALUE byte-array ceiling on
tables with millions of partitions (events-v1: ~6.7M).
Design:
* HoodieCleanConfig: new CLEANER_PLAN_PARTITIONS_BATCH_SIZE property
and builder method. HoodieWriteConfig: matching getter.
* CleanPlanActionExecutor: added BatchOverride carrying both the
partition slice AND the pinned earliestInstant.
* BaseHoodieTableServiceClient: at the top of clean(...), when the
new config is > 0, delegate to a new cleanInBatches() path.
* HoodieCleaner (utility): no change. The new config key is picked up
automatically via --hoodie-conf / withProps, and client.clean()
routes through the modified clean(String, boolean) path.
Non-goals honored: partitionsToClean computation is unchanged, policy
semantics are unchanged, and the incremental-mode/archiver interaction
is untouched. Legacy behavior (batch.size=-1) is preserved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
