venkata91 commented on code in PR #17820:
URL: https://github.com/apache/iceberg/pull/17820#discussion_r4542905176
##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java:
##########
@@ -285,20 +287,24 @@ private void abort(WriterCommitMessage[] messages) {
private DataFileSet files(WriterCommitMessage[] messages) {
DataFileSet files = DataFileSet.create();
+ // every commit path funnels through here, so this is where task write
times are collected
+ WriteDurations durations = WriteDurations.EMPTY;
for (WriterCommitMessage message : messages) {
if (message != null) {
TaskCommit taskCommit = (TaskCommit) message;
files.addAll(Arrays.asList(taskCommit.files()));
+ durations = durations.combine(taskCommit.writeDurations());
Review Comment:
@huaxingao gentle ping on this one when you get a chance. Would you go with
option 1 (keep it on the driver) or option 2 (`writeDuration` as a task
metric), or something else? Happy to update the PR either way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
