nsivabalan commented on issue #20489: URL: https://github.com/apache/hudi/issues/20489#issuecomment-6649901964
Thanks @gudladona for tracking this down. Finding that the per-group payloads multiply across the single union stage's task binary, and backing it with the reproducing test and the heap/JFR numbers, makes the problem very clear. The proposal looks good to us. One minor note on (3)/(4): if we keep `withBulkInsertParallelism`, it would need to come after `withProps(...)` to take effect, since `withProps` does a `putAll` that overwrites it. However, nothing on the clustering write path reads `BULKINSERT_PARALLELISM_VALUE`. Both the row path (`repartitionRecords(inputRecords, numOutputGroups)`) and the RDD path (`SparkBulkInsertHelper.bulkInsert(..., numOutputGroups, ...)`) already pass `numOutputGroups` explicitly. Making it effective would only make configs differ per group and reduce how much can be shared. So dropping the call seems cleaner, and it lets all groups share a single write config. Also, if you plan to take on (6) (writing the group's `RDD<InternalRow>` without the DataFrame round-trip), we'd recommend a separate follow-up PR. It changes the write path more broadly than (1)–(5), which are mechanical sharing changes, and keeping them apart makes each easier to review and safer to land. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
