venkata91 commented on a change in pull request #33896:
URL: https://github.com/apache/spark/pull/33896#discussion_r746929268
##########
File path: core/src/main/scala/org/apache/spark/shuffle/ShuffleBlockPusher.scala
##########
@@ -327,10 +338,32 @@ private[spark] class ShuffleBlockPusher(conf: SparkConf)
extends Logging {
s"stop.")
return false
} else {
+ if (remainingBlocks.isEmpty && pushRequests.isEmpty &&
deferredPushRequests.isEmpty) {
Review comment:
@Ngone51 Wouldn't we add the blocks to `remainingBlocks` here when there
are multiple requests andwhen the `onBlockPushSuccess` is triggered for the
first request, we would remove the blocks of the first request from
`remainingBlocks` but we have [already enqueued the `blockIds` of the 2nd
request](https://github.com/apache/spark/blob/4801b3fb06fbe4c434397aff29c52ed083fb8006/core/src/main/scala/org/apache/spark/shuffle/ShuffleBlockPusher.scala#L219)
as well right? so `updateStateAndCheckIfPushMore` is still referencing the
same `Set` so it won't be empty right? Am I missing something here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
