rangareddy commented on issue #4682: URL: https://github.com/apache/hudi/issues/4682#issuecomment-5102163293
This issue was reviewed as part of the JIRA-migrated backlog triage. Findings: Worth recording what this thread actually established, because part of it was a misdiagnosis. **The 4 / 20 / 100 task counts were not poor parallelization.** That stage's parallelism is set to the number of partitions in the call, by construction: `context.mapToPair(partitionPaths, ..., partitionPaths.size())` (`UpsertPartitioner.java:287`). A stage with 4 tasks means the batch touched 4 partitions. Also, the lookup was already an RDD `mapToPair` back in 0.10.0 -- it was never serial -- so "parallelize it" was not the missing fix. What genuinely changed after 0.10.0 is the addition of a short-circuit: when `hoodie.parquet.small.file.limit` is `0`, the lookup is now skipped entirely (absent at `release-0.10.0`, present from `release-0.12.0`). **The regression was resolved in-thread by configuration.** @tjtoll reported on 2022-03-28 that removing `hoodie.copyonwrite.record.size.estimate` and switching from a timestamp (`yyyyMM`) partition to a range partition on the auto-incrementing record key restored 0.8-level performance, and posted the before/after config on 2022-03-29. A stale or wrong `record.size.estimate` is a good candidate for the original symptom, since it feeds the bin-packing decisions that drive small-file handling. The original 0.8 → 0.10 regression was never bisected to a commit, the 0.11 retest was abandoned after unrelated Glue hive-sync errors, and @ChiehFu never followed up. There has been no substantive discussion since 2022-09-14. Closing as stale. The same stages are still being discussed on current versions in #2620, which is being kept open as the tracking issue for a performance pass -- please add current-version numbers there if you still see this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
