wangzhigang1999 opened a new issue, #10834: URL: https://github.com/apache/paimon/issues/10834
### Search before asking - [x] I searched in the issues and found no matching report for these two metric defects. ### Paimon version 2.2-SNAPSHOT, master at `ed8ecdc23177606a57d7d21a81ee12b4512038f0`. ### Compute Engine Spark 3.5.8 / Scala 2.12.18 and Spark 4.1.2 / Scala 2.13.17, with JDK 17. ### Minimal reproduce step The two defects can be reproduced at the reader level without external storage: 1. Use a `RecordReader` whose first `readBatch()` waits 25 ms and returns a non-empty batch. Construct a `PaimonRecordReaderIterator` around it and inspect `readBatchTimeMs`. The result is 0 ms: the constructor calls `readBatch()` before initializing `readBatchTimeNs` to zero, which discards the elapsed time. 2. Move that initialization before the first `readBatch()` to isolate the second defect. Create a `PaimonPartitionReader` with a single split backed by the delayed reader, consume one row, and read its `readBatchTime` metric before exhausting or closing the split. The result is still 0 ms, even though the active iterator now has a nonzero elapsed time. The metric getter only includes completed splits. The first defect is visible in the [iterator initialization](https://github.com/apache/paimon/blob/ed8ecdc23177606a57d7d21a81ee12b4512038f0/paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/PaimonRecordReaderIterator.scala#L80); the second is in the [partition reader's metric getter](https://github.com/apache/paimon/blob/ed8ecdc23177606a57d7d21a81ee12b4512038f0/paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/PaimonPartitionReader.scala#L137). ### What doesn't meet your expectations? The first batch should contribute to `readBatchTime`. A metric snapshot should include both completed splits and the active split. Spark collects custom task metrics before closing the partition reader on task completion. If a query such as `LIMIT` stops within a split, accounting for that split only during `close()` is too late for the final metric snapshot. Depending on the scan, the reported value can be zero or lower than the recorded batch time. ### Anything else? The expected behavior retains the existing `readBatch()` timing scope and per-split millisecond conversion. This concerns metric accuracy; query elapsed time includes work outside that scope. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
