Z1Wu commented on code in PR #3115:
URL: https://github.com/apache/celeborn/pull/3115#discussion_r1969745626
##########
client/src/main/java/org/apache/celeborn/client/read/DfsPartitionReader.java:
##########
@@ -122,7 +126,7 @@ public DfsPartitionReader(
e);
}
- if (endMapIndex != Integer.MAX_VALUE) {
+ if (endMapIndex != Integer.MAX_VALUE && endMapIndex != -1) {
Review Comment:
In my understanding, currently celeborn supports both sort-based
skew-handling and the sort-free skew-handling introduced in
https://github.com/apache/celeborn/pull/2373. If we change the condition to
`startMapIndex > endMapIndex`, then it will no longer support sort-based
skew-handling at all.
There are some other sort-based skew-handling code in current codebase, such
as `PartitionFilesSorter`. If we want to completely remove the code related to
sort-based skew-handling, would it be more appropriate to do it in a new PR
after https://github.com/apache/celeborn/pull/3118 is merged into the main
branch?
Please correct me if there's anything I haven't understood correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
