Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/17480#discussion_r110802758
--- Diff:
core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala ---
@@ -249,7 +249,14 @@ private[spark] class ExecutorAllocationManager(
* yarn-client mode when AM re-registers after a failure.
*/
def reset(): Unit = synchronized {
- initializing = true
+ /**
+ * When some tasks need to be scheduled and initial executor = 0,
resetting the initializing
+ * field may cause it to not be set to false in yarn.
+ * SPARK-20079: https://issues.apache.org/jira/browse/SPARK-20079
+ */
+ if (maxNumExecutorsNeeded() == 0) {
+ initializing = true
--- End diff --
@vanzin sorry I think I didn't explain well.
If this flag `initializing` is set to false during initialization,
`updateAndSyncNumExecutorsTarget` will recalculate the required executor number
and ramp down the executors if there's no job in the current time. And then if
first job is submitted, it still requires to ramp up executors to meet the
requirement.
For the AM restart scenario I think it is similar during initializing. One
exception is the scenario mentioned here, for the case here should ramp up soon
to meet the requirement.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
