sunchao commented on code in PR #3784:
URL: https://github.com/apache/celeborn/pull/3784#discussion_r3732822852
##########
worker/src/main/java/org/apache/celeborn/service/deploy/worker/storage/PartitionFilesSorter.java:
##########
@@ -69,6 +69,7 @@
public class PartitionFilesSorter extends ShuffleRecoverHelper {
private static final Logger logger =
LoggerFactory.getLogger(PartitionFilesSorter.class);
+ private static final int SORTED_FILE_RESOLVE_THREADS = 4;
Review Comment:
Done
##########
worker/src/main/java/org/apache/celeborn/service/deploy/worker/storage/PartitionFilesSorter.java:
##########
@@ -314,6 +328,168 @@ public FileInfo getSortedFileInfo(
}
}
+ public CompletableFuture<FileInfo> getSortedFileInfoAsync(
+ String shuffleKey, String fileName, FileInfo fileInfo, int
startMapIndex, int endMapIndex) {
+ if (shutdown) {
+ return failedSortedFileInfo(new IOException("Partition sorter is
closed."));
+ }
+ if (fileInfo instanceof MemoryFileInfo) {
+ try {
+ return CompletableFuture.completedFuture(
+ getSortedFileInfo(shuffleKey, fileName, fileInfo, startMapIndex,
endMapIndex));
+ } catch (IOException e) {
+ return failedSortedFileInfo(e);
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
