venkata91 commented on code in PR #17820:
URL: https://github.com/apache/iceberg/pull/17820#discussion_r4570887782


##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java:
##########
@@ -285,20 +287,24 @@ private void abort(WriterCommitMessage[] messages) {
 
   private DataFileSet files(WriterCommitMessage[] messages) {
     DataFileSet files = DataFileSet.create();
+    // every commit path funnels through here, so this is where task write 
times are collected
+    WriteDurations durations = WriteDurations.EMPTY;
 
     for (WriterCommitMessage message : messages) {
       if (message != null) {
         TaskCommit taskCommit = (TaskCommit) message;
         files.addAll(Arrays.asList(taskCommit.files()));
+        durations = durations.combine(taskCommit.writeDurations());

Review Comment:
   Thanks for pushing on this, you were right to ask. WritersBenchmark actually 
doesn't go through SparkWrite's DataWriter (it calls 
ClusteredDataWriter/FanoutDataWriter directly), so I wrote a small local JMH 
that drives SparkWrite's own writers with the same 2.5M-row Parquet setup, and 
compared the PR against the PR minus the per-row timer.
   
   The overhead is visible on the unpartitioned path. It was 4% to 12% slower 
across four separate runs, about 60 to 190 ns per row, and the no-timer variant 
matched main. On the partitioned paths the difference was within noise on my 
(pretty busy) box. The lambda mostly gets scalar-replaced, but in the fanout 
writer some JVM forks did allocate it per row (+16 to 40 B/row). And it's the 
nanoTime calls themselves rather than the lambda: an inline nanoTime version 
without the lambda was even slower. A nanoTime pair is ~49 ns here.
   
   So I'll take per-row timing out of write(). I'm thinking of timing at the 
output stream instead (wrap the FileIO we give OutputFileFactory), so we only 
read the clock per buffer flush, and keeping the close timing. That still 
separates storage time from shuffle/sort, which is what I was after. Does that 
sound reasonable?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to