venkata91 commented on code in PR #17820:
URL: https://github.com/apache/iceberg/pull/17820#discussion_r4570887782
##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java:
##########
@@ -285,20 +287,24 @@ private void abort(WriterCommitMessage[] messages) {
private DataFileSet files(WriterCommitMessage[] messages) {
DataFileSet files = DataFileSet.create();
+ // every commit path funnels through here, so this is where task write
times are collected
+ WriteDurations durations = WriteDurations.EMPTY;
for (WriterCommitMessage message : messages) {
if (message != null) {
TaskCommit taskCommit = (TaskCommit) message;
files.addAll(Arrays.asList(taskCommit.files()));
+ durations = durations.combine(taskCommit.writeDurations());
Review Comment:
Thanks for pushing on this, you were right to ask. WritersBenchmark actually
doesn't go through SparkWrite's DataWriter (it calls
ClusteredDataWriter/FanoutDataWriter directly), so I wrote a small local JMH
that drives SparkWrite's own writers with the same 2.5M-row Parquet setup, and
compared the PR against the PR minus the per-row timer.
The overhead is visible on the unpartitioned path. It was 4% to 12% slower
across four separate runs, about 60 to 190 ns per row, and the no-timer variant
matched main. On the partitioned paths the difference was within noise on my
(pretty busy) box. The lambda mostly gets scalar-replaced, but in the fanout
writer some JVM forks did allocate it per row (+16 to 40 B/row). And it's the
nanoTime calls themselves rather than the lambda: an inline nanoTime version
without the lambda was even slower. A nanoTime pair is ~49 ns here.
So I'll take per-row timing out of write(). I'm thinking of timing at the
output stream instead (wrap the FileIO we give OutputFileFactory), so we only
read the clock per buffer flush, and keeping the close timing. That still
separates storage time from shuffle/sort, which is what I was after. Does that
sound reasonable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]