mridulm commented on code in PR #37638:
URL: https://github.com/apache/spark/pull/37638#discussion_r1050430333
##########
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java:
##########
@@ -1197,15 +1230,15 @@ public void onData(String streamId, ByteBuffer buf)
throws IOException {
appShuffleInfo.shuffles.get(partitionInfo.appAttemptShuffleMergeId.shuffleId);
if (isStale(info,
partitionInfo.appAttemptShuffleMergeId.shuffleMergeId) ||
Review Comment:
I would look at these metrics as a means for identifying the effectiveness
of push based shuffle, what is impacting the inefficiencies (how many stale
blocks, how many late blocks, how that distribution is changing with time), etc.
These will correlate in some way to overall impact on when push based
shuffle was used, when normal shuffle was used, and what the network and task
impact of those are.
Do let me know if you have thoughts !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
