cloud-fan commented on a change in pull request #32287:
URL: https://github.com/apache/spark/pull/32287#discussion_r623658038
##########
File path:
core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala
##########
@@ -862,6 +967,31 @@ private class ShuffleFetchCompletionListener(var data:
ShuffleBlockFetcherIterat
private[storage]
object ShuffleBlockFetcherIterator {
+ /**
+ * A flag which indicates whether the Netty OOM error has raised during
shuffle.
+ * If true, unless there's no in-flight fetch requests, all the pending
shuffle
+ * fetch requests will be deferred until the flag is unset (whenever there's
a
+ * complete fetch request).
+ */
+ val isNettyOOMOnShuffle = new AtomicBoolean(false)
+
+ /**
+ * The least free memory (200M - the default value of
`maxReqSizeShuffleToMem`) that
+ * Netty should reserve before unset the `isNettyOOMOnShuffle`.
+ */
+ val freeNettyMemoryLowerBound = 200 * 1024 * 1024
Review comment:
can we read the value of conf `MAX_REMOTE_BLOCK_SIZE_FETCH_TO_MEM`
instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
