danny0405 opened a new pull request, #20024: URL: https://github.com/apache/hudi/pull/20024
### Describe the issue this Pull Request addresses Compaction planning currently passes the selected partition count as engine parallelism. For tables with many partitions, this can create excessive concurrent file-system-view requests and transient file-group/file-slice allocations, increasing planning memory pressure and the risk of out-of-memory failures. In Flink, the value also determines the dedicated ForkJoinPool parallelism. ### Summary and Changelog - Add `hoodie.compaction.plan.parallelism`, defaulting to 200, with a builder method and write-config accessor. Reject non-positive values when building the write configuration. - Bound the shared compaction and log-compaction planner's partition scan by `min(selected partition count, configured parallelism)` after strategy filtering. All selected partitions remain eligible for scanning. - Add configuration tests for defaults, builder/property overrides, and invalid values, plus planner tests for the cap, smaller partition counts, filtered partitions, and empty selections. No code was copied from external sources. ### Impact Adds a public configuration and additive configuration APIs. Spark and Flink compaction planning use at most the configured engine parallelism, which can reduce concurrent FS-view load but increase scheduling latency for large tables. Tables selecting fewer than 200 partitions retain their default planning parallelism. This does not bound retained FS-view cache size or change compaction execution parallelism, storage formats, or clustering planning. `HoodieLocalEngineContext` currently ignores the parallelism argument, so its thread concurrency is unaffected. ### Risk Level low The change only limits the parallelism passed to the engine; partition selection and operation construction are unchanged. Large tables may need to tune the new setting to balance planning latency and resource use. Validation passed: 51 tests across `TestHoodieWriteConfig` and `TestBaseHoodieCompactionPlanGenerator`, with zero failures/errors/skips, plus Checkstyle and Apache RAT license checks. ```bash mvn -pl hudi-client/hudi-client-common -Dtest=TestHoodieWriteConfig,TestBaseHoodieCompactionPlanGenerator -DfailIfNoTests=false test ``` The planner regression tests verify the submitted parallelism and that every selected partition is scanned using a local engine spy; they are not a distributed concurrency or memory benchmark. ### Documentation Update Added the configuration description, default, version, and tuning guidance to `HoodieCompactionConfig` for the generated configuration reference. The website configuration reference will need to include this property when regenerated for the release; no website files are changed in this PR. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
