dongjoon-hyun commented on code in PR #58054:
URL: https://github.com/apache/spark/pull/58054#discussion_r3806531571
##########
core/src/main/scala/org/apache/spark/SparkContext.scala:
##########
@@ -2075,6 +2075,159 @@ class SparkContext(config: SparkConf) extends Logging {
}
}
+ // Whether the executors are held via `holdExecutors()`, and, when dynamic
allocation is
+ // disabled, the number of executors to restore on `resumeExecutors()`.
+ @volatile private var _executorsHeld: Boolean = false
+ private var heldNumExecutors: Int = 0
+
+ /**
+ * Whether `holdExecutors()` is supported in the current deployment. It
requires a scheduler
+ * backend that can adjust the number of executors, decommission support,
and shuffle data
+ * kept outside the executors: either an external shuffle service or a
`ShuffleDataIO` with
+ * reliable storage.
+ */
+ private[spark] def executorHoldSupported: Boolean = {
+ schedulerBackend.isInstanceOf[ExecutorAllocationClient] &&
+ (conf.get(SHUFFLE_SERVICE_ENABLED) ||
shuffleDriverComponents.supportsReliableStorage()) &&
Review Comment:
Fixed the description: the requirement is now stated as a disjunction
(external shuffle service or a `ShuffleDataIO` with reliable storage), and the
recomputation sentence is qualified to shuffle output with an explicit note
that cached blocks are recomputed. Also fixed the `1 executors` plural in the
state diagram.
##########
docs/configuration.md:
##########
@@ -1605,6 +1605,19 @@ Apart from these, the following properties are also
available, and may be useful
</td>
<td>1.0.0</td>
</tr>
+<tr>
+ <td><code>spark.ui.holdEnabled</code></td>
Review Comment:
Added in 137ebdb to the Jobs Tab section of `web-ui.md`, based on your
draft, including the note that the control does not render when the
prerequisites are not met.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
