zhouyejoe commented on code in PR #36165:
URL: https://github.com/apache/spark/pull/36165#discussion_r1030723363
##########
core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala:
##########
@@ -282,6 +280,17 @@ final class ShuffleBlockFetcherIterator(
}
}
+ @inline def updateMergedReqsDuration(wasReqForMergedChunks: Boolean =
false): Unit = {
+ if (remainingBlocks.isEmpty) {
+ val durationMs = TimeUnit.NANOSECONDS.toMillis(clock.nanoTime() -
requestStartTime)
+ if (wasReqForMergedChunks) {
+ shuffleMetrics.incRemoteMergedReqsDuration(durationMs)
+ } else {
+ shuffleMetrics.incRemoteReqsDuration(durationMs)
+ }
Review Comment:
+1 on this point, the merged remote request duration should be counted into
the total remote request duration, following the same logic for the
local/remote bytes read.
@otterc WDTY?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
