zhouyejoe commented on a change in pull request #32007:
URL: https://github.com/apache/spark/pull/32007#discussion_r637355653
##########
File path:
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -117,7 +116,7 @@ public ShuffleIndexInformation load(File file) throws
IOException {
private AppShufflePartitionInfo getOrCreateAppShufflePartitionInfo(
AppShuffleId appShuffleId,
int reduceId) {
- File dataFile = getMergedShuffleDataFile(appShuffleId, reduceId);
+ File dataFile = getMergedShuffleDataFile(appShuffleId.appId,
appShuffleId.shuffleId, reduceId);
Review comment:
Added AttemptID check here. If the attemptID is different with the one
recorded in AppAttemptPathInfo, it will return null and will take this block as
a late block, so it will be ignored.
##########
File path:
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -991,22 +1023,26 @@ int getNumIOExceptions() {
/**
* Wraps all the information related to the merge directory of an
application.
*/
- private static class AppPathsInfo {
+ private static class AppAttemptPathsInfo {
+ private final int attemptId;
private final String[] activeLocalDirs;
private final int subDirsPerLocalDir;
- private AppPathsInfo(
+ private AppAttemptPathsInfo(
String appId,
+ int attemptId,
Review comment:
Changed to AppAttemptShuffleId
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
