[GitHub] [spark] Ngone51 commented on a change in pull request #33078: [SPARK-35546][Shuffle] Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the state in a better way

[GitBox]

Ngone51 commented on a change in pull request #33078:
URL: https://github.com/apache/spark/pull/33078#discussion_r660706937



##########
File path: 
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -73,27 +73,31 @@
 public class RemoteBlockPushResolver implements MergedShuffleFileManager {
 
   private static final Logger logger = 
LoggerFactory.getLogger(RemoteBlockPushResolver.class);
-  @VisibleForTesting
-  static final String MERGE_MANAGER_DIR = "merge_manager";
+
   public static final String MERGED_SHUFFLE_FILE_NAME_PREFIX = "shuffleMerged";
 
-  private final ConcurrentMap<String, AppPathsInfo> appsPathInfo;
-  private final ConcurrentMap<AppShuffleId, Map<Integer, 
AppShufflePartitionInfo>> partitions;
+  /**
+   *  A concurrent hashmap where the key is the applicationId, and the value 
includes
+   *  all the merged shuffle information for this application. AppShuffleInfo 
stores
+   *  the application attemptId, merged shuffle local directories and the 
metadata
+   *  for actively being merged shuffle partitions.

Review comment:
       one ident?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org