venkata91 commented on a change in pull request #33034:
URL: https://github.com/apache/spark/pull/33034#discussion_r676161785
##########
File path:
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -191,36 +208,28 @@ private AbstractFetchShuffleBlocks
createFetchShuffleMsgAndBuildBlockIds(
}
}
assert(blockIdIndex == this.blockIds.length);
- Set<Number> primaryIds = primaryIdToBlocksInfo.keySet();
- if (!areMergedChunks) {
- long[] mapIds = Longs.toArray(primaryIds);
- return new FetchShuffleBlocks(
- appId, execId, shuffleId, mapIds, secondaryIdsArray,
batchFetchEnabled);
- } else {
- int[] reduceIds = Ints.toArray(primaryIds);
- return new FetchShuffleBlockChunks(appId, execId, shuffleId, reduceIds,
secondaryIdsArray);
- }
+ return secondaryIds;
}
- /** Split the shuffleBlockId and return shuffleId, mapId and reduceIds. */
+ /**
+ * Split the blockId and return accordingly
+ * shuffleChunk - return shuffleId, shuffleMergeId, reduceId and chunkIds
+ * shuffle block - return shuffleId, mapId, reduceId
+ * shuffle batch block - return shuffleId, mapId, begin reduceId and end
reduceId
+ */
private String[] splitBlockId(String blockId) {
String[] blockIdParts = blockId.split("_");
// For batch block id, the format contains shuffleId, mapId, begin
reduceId, end reduceId.
- // For single block id, the format contains shuffleId, mapId, educeId.
- // For single block chunk id, the format contains shuffleId, reduceId,
chunkId.
+ // For single block id, the format contains shuffleId, mapId, reduceId.
+ // For single block chunk id, the format contains shuffleId,
shuffleMergeId, reduceId, chunkId.
if (blockIdParts.length < 4 || blockIdParts.length > 5) {
Review comment:
Original shuffleBlock is only of length 4 right? `shuffle_1_1_1`? This
case `// For single block id, the format contains shuffleId, mapId, reduceId.`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
