cloud-fan commented on a change in pull request #32287:
URL: https://github.com/apache/spark/pull/32287#discussion_r627961194
##########
File path:
core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala
##########
@@ -862,6 +962,25 @@ private class ShuffleFetchCompletionListener(var data:
ShuffleBlockFetcherIterat
private[storage]
object ShuffleBlockFetcherIterator {
+ /**
+ * A flag which indicates whether the Netty OOM error has raised during
shuffle.
+ * If true, unless there's no in-flight fetch requests, all the pending
shuffle
+ * fetch requests will be deferred until the flag is unset (whenever there's
a
+ * complete fetch request).
+ */
+ val isNettyOOMOnShuffle = new AtomicBoolean(false)
+
+ /**
+ * The max number of a block could retry due to Netty OOM before throwing
the fetch failure.
+ */
+ val maxAttemptsOnNettyOOM: Int =
SparkEnv.get.conf.get(SHUFFLE_MAX_ATTEMPTS_ON_NETTY_OOM)
Review comment:
It's a bit fragile to read a spark conf in `object`. Shall we read it
when we instantiate `ShuffleBlockFetcherIterator`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
