venkata91 commented on code in PR #17820:
URL: https://github.com/apache/iceberg/pull/17820#discussion_r4580698335
##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java:
##########
@@ -285,20 +287,24 @@ private void abort(WriterCommitMessage[] messages) {
private DataFileSet files(WriterCommitMessage[] messages) {
DataFileSet files = DataFileSet.create();
+ // every commit path funnels through here, so this is where task write
times are collected
+ WriteDurations durations = WriteDurations.EMPTY;
for (WriterCommitMessage message : messages) {
if (message != null) {
TaskCommit taskCommit = (TaskCommit) message;
files.addAll(Arrays.asList(taskCommit.files()));
+ durations = durations.combine(taskCommit.writeDurations());
Review Comment:
I took out the per-row timing from `write()` everywhere and kept the close
timing.
Instead, `TimedFileIO` goes into `OutputFileFactory` through `ioSupplier`
and times the storage calls (create, write, flush, close) on each output
stream, so the clock is read per stream call, not per row. It's scoped to
`HadoopFileIO`: for a `HadoopOutputFile` it wraps the file's `FileSystem`,
because Parquet and ORC write through the `FileSystem` and not
`OutputFile.create()`. That covers Parquet, ORC, Avro and DV files there. With
other FileIOs the files are written untimed and the storage metrics read 0.
Parquet also looked the `FileSystem` up again from the path, which I fixed
separately in #18947. This PR carries that commit until it merges.
Spark reads task metrics before `commit()`, which is where files get closed
and most uploads finish, so the old `total (min, med, max)` string really
showed `total (0, 0, total)`. Now the writer is wrapped to send its times in
the commit message, and the driver reports total storage write time plus the
min, median and max across tasks as separate metrics. Each task also logs its
storage time and bytes at INFO, so you can find the slow one.
On a local JMH (SparkWrite unpartitioned, 2.5M rows, 12 interleaved rounds
against no timer) it's +2.2% on average with an SE of 2.3%, and half the rounds
came out faster with the timer. Allocation per row is the same. That write
makes about 29K timed calls, so I'd expect around 1.4 ms of clock reads in a
3.7 s write, and I think the rest is noise on that box. The per-row version was
4% to 12% slower.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]