rangareddy commented on issue #3077: URL: https://github.com/apache/hudi/issues/3077#issuecomment-5103004459
This issue was reviewed as part of the JIRA-migrated backlog triage. Findings: This is a tuning thread rather than an identified defect, and the workload description points at the answer that was never actually tested. With **99%+ inserts and under 1% updates**, `upsert` is the wrong operation -- every write pays for index lookup and small-file packing to service a fraction of a percent of records. `bulk_insert` for the initial load and `insert` for subsequent batches, with **clustering** scheduled separately to consolidate file sizes, is the configuration for this shape. MOR was tried and did not help, which is expected: MOR optimises *updates*, and there are almost none here. On the specific stage that dominated: its parallelism is the number of partitions the batch touches, by construction (`UpsertPartitioner.java:287`), so with 5 partitions it will always be a 5-task stage regardless of `hoodie.upsert.shuffle.parallelism`. That explains why raising parallelism had little effect. The last round of suggestions (2022-09-13: disable small-file handling, enable clustering, raise max file size to ~250MB) was never followed up, and there has been no substantive discussion since. Closing as stale -- the report is against an 0.8-era write path that has been substantially rewritten, so re-measuring it would not be meaningful. The same two stages are still being discussed on recent versions in #2620, which is being kept open as the tracking issue for a performance pass. If large-batch upsert latency is still a problem for you on 1.0.x / 1.1.x / 1.2.0, please add your numbers there or open a fresh issue with the Hudi version, write operation, and Spark UI stage timings. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
