rangareddy commented on issue #3324: URL: https://github.com/apache/hudi/issues/3324#issuecomment-5102130257
This issue was reviewed as part of the JIRA-migrated backlog triage. Findings: Two things in this thread put it out of reach of a reproducible upstream investigation. First, the useful isolation you did yourself: throughput tracked the **number of columns projected** (~1000) rather than the row count. That is a wide-schema write cost, and it is worth noting it is largely paid outside Hudi's control path -- Avro/parquet schema handling and per-record conversion scale with column count regardless of how few rows are in the micro-batch. Second, on 2021-09-14 you noted the behaviour appeared to be caused by an Amazon build of Spark, and that it was triggered by particular Hudi + Amazon-Spark combinations. That makes it not reproducible against upstream Hudi on a vanilla Spark, which is where any fix would have to be validated. Requests to retest on 0.12 (2022-09-12) and a pointer to multi-writer support for the concurrency question (2022-11-02) went unanswered, and there has been no activity since. Closing as stale. If wide-schema streaming writes are still slow for you, please open a fresh issue against a recent Hudi (1.0.x / 1.1.x / 1.2.0) on a vanilla Spark build, with the column count, the micro-batch size, and Spark UI stage timings -- ideally with `hoodie.datasource.write.row.writer.enable=true` for `bulk_insert`, which avoids the per-record Avro conversion that dominates wide schemas. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
