rangareddy commented on issue #17386: URL: https://github.com/apache/hudi/issues/17386#issuecomment-5351242500
This issue was reviewed as part of the JIRA-migrated backlog triage (HUDI-9111). **Findings: not done, and there is a concrete inconsistency to point at.** In `hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/common/HoodieSparkEngineContext.java`, the `parallelism` argument is applied to `reduceByKey` in only one of the three reduce paths: | method | line | `reduceByKey` call | parallelism honoured | | --- | --- | --- | --- | | `mapToPairAndReduceByKey(List, ..., int parallelism)` | `:148` | `:154` `.reduceByKey(reduceFunc::apply)` | **no** | | `mapToPairAndReduceByKey(HoodieData, ..., int parallelism)` | `:160` | `:168` `.reduceByKey(reduceFunc::apply, parallelism)` | yes | | `reduceByKey(List<Pair>, ..., int parallelism)` | `:174` | `:177` `.reduceByKey(reduceFunc::apply)` | **no** | In the two "no" cases the `parallelism` argument is still used for the initial `parallelize(...)`, so the caller's intent is partially honoured and then silently dropped at the shuffle, which falls back to Spark's default partitioner. That is the kind of thing that looks configured but is not, and it matches the concern raised in [the review comment on #12874](https://github.com/apache/hudi/pull/12874#discussion_r1966456842). Two of these are likely a one-line fix each; the value of the ticket is deciding the convention deliberately (should every API taking `parallelism` pass it through to every shuffle?) and then applying it across the class, rather than patching the two sites in isolation. Keeping this open. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
