mridulm commented on a change in pull request #33034:
URL: https://github.com/apache/spark/pull/33034#discussion_r677993932
##########
File path:
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -824,12 +943,34 @@ AppShufflePartitionInfo getPartitionInfo() {
}
}
+ public static class AppShuffleMergePartitionsInfo {
+ private final int shuffleMergeId;
+ private Map<Integer, AppShufflePartitionInfo> shuffleMergePartitions;
+
+ public AppShuffleMergePartitionsInfo(
+ int shuffleMergeId) {
+ this.shuffleMergeId = shuffleMergeId;
+ this.shuffleMergePartitions = new ConcurrentHashMap<>();
+ }
+
+ @VisibleForTesting
+ public Map<Integer, AppShufflePartitionInfo> getShuffleMergePartitions() {
+ return shuffleMergePartitions;
+ }
+
+ public void markIndeterminateShuffleFinalized() {
+ this.shuffleMergePartitions =
RemoteBlockPushResolver.INDETERMINATE_SHUFFLE_FINALIZED;
+ }
+ }
Review comment:
As mentioned above, let us make this class immutable.
Also, add a short javadoc for the class.
```suggestion
private final int shuffleMergeId;
private final Map<Integer, AppShufflePartitionInfo>
shuffleMergePartitions;
public AppShuffleMergePartitionsInfo(int shuffleMergeId, boolean
shuffleFinalized) {
this.shuffleMergeId = shuffleMergeId;
this.shuffleMergePartitions = shuffleFinalized ?
RemoteBlockPushResolver.INDETERMINATE_SHUFFLE_FINALIZED : new
ConcurrentHashMap<>();
}
@VisibleForTesting
public Map<Integer, AppShufflePartitionInfo> getShuffleMergePartitions()
{
return shuffleMergePartitions;
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
