Victsm commented on a change in pull request #33613:
URL: https://github.com/apache/spark/pull/33613#discussion_r683644741
##########
File path:
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -385,14 +397,24 @@ public StreamCallbackWithID
receiveBlockDataAsStream(PushBlockStream msg) {
+ "with the current attempt id %s stored in shuffle service for
application %s",
msg.appAttemptId, appShuffleInfo.attemptId, msg.appId));
}
+ // Use string concatenation here to avoid the overhead with String.format
on every
+ // pushed block.
+ final String streamId = OneForOneBlockPusher.SHUFFLE_PUSH_BLOCK_PREFIX +
"_"
+ + msg.shuffleId + "_" + msg.shuffleMergeId + "_" + msg.mapIndex + "_" +
msg.reduceId;
// Retrieve merged shuffle file metadata
AppShufflePartitionInfo partitionInfoBeforeCheck;
+ boolean isStaleBlock = false;
+ boolean isTooLate = false;
try {
partitionInfoBeforeCheck =
getOrCreateAppShufflePartitionInfo(appShuffleInfo, msg.shuffleId,
msg.shuffleMergeId, msg.reduceId);
- } catch(StaleBlockPushException sbp) {
+ isTooLate = partitionInfoBeforeCheck == null;
+ } catch(BlockPushNonFatalFailure bpf) {
Review comment:
Unable to create these files seems like a more disrupting issue.
It could be a temporary glitch, but it could also be due to issues like disk
failures or disk being full, which will just fail all other block pushes even
for different partitions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
