Ngone51 commented on a change in pull request #32287:
URL: https://github.com/apache/spark/pull/32287#discussion_r625781682
##########
File path:
core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala
##########
@@ -245,9 +253,21 @@ final class ShuffleBlockFetcherIterator(
case FetchBlockInfo(blockId, size, mapIndex) => (blockId.toString,
(size, mapIndex))
}.toMap
val remainingBlocks = new HashSet[String]() ++= infoMap.keys
+ val deferredBlocks = new ArrayBuffer[String]()
val blockIds = req.blocks.map(_.blockId.toString)
val address = req.address
+ @inline def enqueueDeferredFetchRequestIfNecessary(): Unit = {
+ if (remainingBlocks.isEmpty && deferredBlocks.nonEmpty) {
+ val blocks = deferredBlocks.map { blockId =>
+ val (size, mapIndex) = infoMap(blockId)
+ FetchBlockInfo(BlockId(blockId), size, mapIndex)
+ }
+ results.put(DeferFetchRequestResult(FetchRequest(address,
blocks.toSeq)))
+ deferredBlocks.clear()
+ }
+ }
Review comment:
So after #32287 (comment), I have changed the unset condition to
`freeDirectorMemory > maxReqSizeShuffleToMem (200M by default)`, which I think
is already very strict. So, it should avoid the issue you mentioned in #32287
(comment).
> Another simple way could be to modify isRemoteBlockFetchable such that
after this iterator has seen an OOM, it will also check bytesInFlight +
fetchReqQueue.front.size < freeDirectorMemory?
Do you mean check bytesInFlight + fetchReqQueue.front.size <
freeDirectorMemory for all the cases or only when `isNettyOOMOnShuffle=true`?
If you also want to check when `isNettyOOMOnShuffle=false`, I'd like to mention
that block size is not equal to the consumed memory size of Netty. As you know,
blocks that bigger than `maxReqSizeShuffleToMem` would be stored on disk.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
