rangareddy commented on issue #5481: URL: https://github.com/apache/hudi/issues/5481#issuecomment-5101003043
This issue was reviewed as part of the JIRA-migrated backlog triage. Findings: This is a performance/tuning report (executor OOM, exit 137, on a ~50GB batch reload into a ~75GB table on Hudi 0.10.0 / Spark 3.1.1 / EMR 6.3.1), not a localized Hudi defect. Concrete tuning guidance was provided on this thread -- setting parallelism from input size (roughly `input size / 100MB` for `spark.default.parallelism`, `spark.sql.shuffle.partitions` and `hoodie.upsert.shuffle.parallelism`), raising `spark.kryoserializer.buffer.max` to `1024m`, and lowering `spark.memory.fraction` / `spark.memory.storageFraction` to trade speed for stability -- along with the [Tuning Guide](https://cwiki.apache.org/confluence/display/HUDI/Tuning+Guide). Follow-ups on 2022-06-29 and 2022-11-04 asking whether those helped went unanswered, and there has been no substantive discussion since 2022-11-04. Because no Hudi-side defect was ever isolated, there is nothing here that a performance validation pass could reproduce or regress against -- and the write and index paths have changed substantially since 0.10.0 (record-level index, and a `FileGroupReader`-based merge path), so a measurement from 0.10.0 would not carry over. Closing as stale. If large-batch upserts are still a bottleneck for you on a recent version (1.0.x / 1.1.x / 1.2.0), please open a fresh issue with the Hudi version, the index type in use, the write configs, input and table sizes, and the Spark UI stage timings for the slow stage. A current-version report with those details is something we can act on; consider `RECORD_INDEX` for the upsert-heavy reload case, since index lookup is usually what dominates this shape of job. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
