Victsm commented on a change in pull request #34156:
URL: https://github.com/apache/spark/pull/34156#discussion_r719747636
##########
File path: core/src/main/scala/org/apache/spark/MapOutputTracker.scala
##########
@@ -1448,7 +1448,7 @@ private[spark] object MapOutputTracker extends Logging {
// TODO: improve push based shuffle to read partial merged blocks
satisfying the start/end
// TODO: map indexes
if (mergeStatuses.exists(_.nonEmpty) && startMapIndex == 0
- && endMapIndex == mapStatuses.length) {
+ && endMapIndex == mapStatuses.length && endPartition - startPartition ==
1) {
Review comment:
@Ngone51
Just take a look at the usage of `PartialMapperPartitionSpec` and
`CoalescedMapperPartitionSpec` for the local shuffle read optimization.
When the local shuffle read optimization kicks in, it indeed makes sense to
not do push-based shuffle for that special case since it's supposed to be a
local shuffle read for changing partitions of the table after AQE-converted bhj.
In these cases, because the reducer for the local shuffle is reading from a
subset of the mappers, push-based shuffle cannot happen in the first place.
However, the current condition for disabling batch fetch in
`BlockStoreShuffleReader` would only check if push-based shuffle is enabled for
the given shuffle dependency, and it will still disable batch fetch even if
push-based shuffle cannot happen because the reducer is not fetching blocks
from all the mappers.
We are evaluating passing this additional information to
`BlockStoreShuffleReader` as well so that we can properly disable batch fetch
only for cases of partition coalesce but not for local shuffle read.
The PR will be updated with this change once validated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
