rangareddy commented on issue #5351:
URL: https://github.com/apache/hudi/issues/5351#issuecomment-5101250244
This issue was reviewed as part of the JIRA-migrated backlog triage.
Findings: This is a performance question rather than an identified defect,
and the measurement that settles it is already on the thread. Writing the same
1.37M-row / 308-column dataset as plain Spark parquet took **568s** against
Hudi's **605s** -- roughly a 6% difference, which leaves no Hudi-specific
overhead to explain. Building from master brought it to 492s.
The expectation gap comes from the other comparison in the thread: a
single-node pandas gzip parquet write at ~2m4s. That is not comparable to a
distributed Spark write of the same data -- the Spark job pays for
partitioning, shuffle, task scheduling and (for Hudi) key generation and index
lookup, on a single `t2.xlarge` in this case. The right baseline for "is Hudi
slow here" is `df.write.format("parquet")` on the same cluster, and by that
baseline it is not.
For the shape of workload described here -- a large initial/immutable load
into a new table -- `bulk_insert` rather than `insert`/`upsert` is the config
that matters, since it skips index lookup and small-file handling entirely; see
the [performance docs](https://hudi.apache.org/docs/performance#bulk-insert).
That suggestion was made on 2022-11-04 and never followed up, and there has
been no substantive discussion since.
Closing as stale, since there is no isolated Hudi defect here for a
performance validation pass to reproduce, and the report is against 0.8.0 -- a
version whose write path has been substantially rewritten since. If write
throughput is still a problem for you on a recent version (1.0.x / 1.1.x /
1.2.0), please open a fresh issue with the Hudi version, the write operation
and configs, the cluster shape, and the Spark UI stage timings, and we can look
at where the time is actually going.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
