venkata91 opened a new pull request, #17820: URL: https://github.com/apache/iceberg/pull/17820
## Problem - Iceberg reports write counts and sizes per commit (`addedDataFiles`, `addedRecords`, ...), but has no timer for how long tasks spent writing, so write throughput cannot be derived from Iceberg metrics alone. ## Fix - Add `writeDuration` (time in the row-level write calls) and `writeCloseDuration` (time closing writers, where buffered rows are flushed and files finished). Nanoseconds, timed per call rather than per row. - Both travel back to the driver in the commit messages. Spark calls `DataWriter.currentMetricsValues()` before `commit()`, so the close, which happens inside `commit()`, cannot be a `CustomTaskMetric`; the commit message keeps both phases on the mechanism every other write metric here already uses. - Aggregate is formatted ns/us/ms/s for the UI, matching `scanDuration` in #17562. - Covers all three write families: `SparkWrite` (append, overwrite, copy-on-write, streaming), `SparkPositionDeltaWrite` (merge-on-read), and `SparkPositionDeletesRewrite` (`DeleteWriter` and `DVWriter`). ## Testing done - `TestWriteTimer`: phases accumulate separately, time is recorded when the write throws, `WriteDurations.combine`. - `TestNanoDurationMetric`: formatting boundaries at 999ns/1us/999us/1ms/999ms/1s. - `TestSparkWriteMetrics`: 7 new tests covering unpartitioned and partitioned inserts, copy-on-write delete, merge-on-read delete, merge-on-read update (unpartitioned and partitioned), and multi-task aggregation. - `TestRewritePositionDeleteFilesMetrics`: rewrite path for both v2 (`DeleteWriter`) and v3 (`DVWriter`). - Assertions discriminate: disabling either timer fails 7 of 9 tests in `TestSparkWriteMetrics`. - Regression sweep, 0 failures: `TestCopyOnWriteDelete` (132), `TestMergeOnReadDelete` (147), `TestMergeOnReadUpdate` (144), `TestRewritePositionDeleteFiles*` (53), `TestPositionDeletesTable` (88), `TestStructuredStreamingRead3` (66), `TestDataFrameWriterV2` (15). `spotlessCheck` and `checkstyle` pass. ## Notes - Scoped to `spark/v4.1`. v4.0 is near-identical and can follow. - v3.5 has no write custom metrics wired at all, so it would need the driver-metrics plumbing backported first. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
