Ngone51 commented on a change in pull request #34156:
URL: https://github.com/apache/spark/pull/34156#discussion_r719576838
##########
File path: core/src/main/scala/org/apache/spark/MapOutputTracker.scala
##########
@@ -1448,7 +1448,7 @@ private[spark] object MapOutputTracker extends Logging {
// TODO: improve push based shuffle to read partial merged blocks
satisfying the start/end
// TODO: map indexes
if (mergeStatuses.exists(_.nonEmpty) && startMapIndex == 0
- && endMapIndex == mapStatuses.length) {
+ && endMapIndex == mapStatuses.length && endPartition - startPartition ==
1) {
Review comment:
> As demonstrated in this PR inside BlockStoreShuffleReader, we can
disable batch fetch when push-based shuffle is enabled and it's an equally
simple fix.
Ok, so you mean to disable batch fetch for all the cases when push-based
shuffle rather than the partition coalesce case only. Then, of course, the
change is also simple.
But in this way, people may hit perf issue for the case like
`PartialMapperPartitionSpec` unless you can guarantee that the improved perf of
PBS is bigger than batch fetch in any cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
