Ngone51 commented on a change in pull request #33617:
URL: https://github.com/apache/spark/pull/33617#discussion_r686822293
##########
File path:
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -514,12 +512,12 @@ public MergeStatuses
finalizeShuffleMerge(FinalizeShuffleMerge msg) {
AppShuffleInfo appShuffleInfo = validateAndGetAppShuffleInfo(msg.appId);
if (appShuffleInfo.attemptId != msg.appAttemptId) {
// If this Block belongs to a former application attempt, it is
considered late,
- // as only the blocks from the current application attempt will be merged
- // TODO: [SPARK-35548] Client should be updated to handle this error.
- throw new IllegalArgumentException(
- String.format("The attempt id %s in this FinalizeShuffleMerge message
does not match "
- + "with the current attempt id %s stored in shuffle service for
application %s",
- msg.appAttemptId, appShuffleInfo.attemptId, msg.appId));
+ // as only the blocks from the current application attempt will be
merged.
+ // No need use callback to return to client now, because the
finalizeShuffleMerge
+ // in DAGScheduler has no retry policy.
Review comment:
https://github.com/apache/spark/blob/bbf988bd73b00d18dd1d443f225b3915a2c4433f/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalBlockHandler.java#L218-L223
Took another look at this, although we don't explicitly use callback here,
it seems like that `BlockPushNonFatalFailure` can't be caught at server side.
So it will still return to `DAGScheduler` with the error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
