rmcyang commented on code in PR #37638:
URL: https://github.com/apache/spark/pull/37638#discussion_r1055805408
##########
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java:
##########
@@ -1904,4 +1941,42 @@ long getPos() {
return pos;
}
}
+
+ /**
+ * A class that wraps all the push-based shuffle service metrics.
+ */
+ static class PushMergeMetrics implements MetricSet {
+ // couldNotFindOpportunityResponses tracks how many times a shuffle block
collided because
+ // of another block for the same reduce partition was being written
+ static final String NO_OPPORTUNITY_RESPONSES_METRIC =
"couldNotFindOpportunityResponses";
+ // tooLateResponses tracks how many times a shuffle block push request is
too late
+ static final String TOO_LATE_RESPONSES_METRIC = "tooLateResponses";
+ // pushedBytesWritten tracks the length of the pushed block data written
to file in bytes
+ static final String PUSHED_BYTES_WRITTEN_METRIC = "pushedBytesWritten";
+ // cachedBlocksBytes tracks the size of the current deferred block parts
buffered in memory.
+ static final String CACHED_BLOCKS_BYTES_METRIC = "cachedBlocksBytes";
Review Comment:
Thanks for the suggestion. Added `deferredBlocks`, PTAL.
##########
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java:
##########
@@ -1197,15 +1230,15 @@ public void onData(String streamId, ByteBuffer buf)
throws IOException {
appShuffleInfo.shuffles.get(partitionInfo.appAttemptShuffleMergeId.shuffleId);
if (isStale(info,
partitionInfo.appAttemptShuffleMergeId.shuffleMergeId) ||
Review Comment:
Make sense. Added `staleBlockPushes`, PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
