Jonathan Kelly created SPARK-10790:
--------------------------------------
Summary: Dynamic Allocation does not request any executors if
first stage needs less than or equal to spark.dynamicAllocation.initialExecutors
Key: SPARK-10790
URL: https://issues.apache.org/jira/browse/SPARK-10790
Project: Spark
Issue Type: Bug
Components: Scheduler
Affects Versions: 1.5.0
Reporter: Jonathan Kelly
Priority: Critical
If you set spark.dynamicAllocation.initialExecutors > 0 (or
spark.dynamicAllocation.minExecutors, since
spark.dynamicAllocation.initialExecutors defaults to
spark.dynamicAllocation.minExecutors), and the number of tasks in the first
stage of your job is less than or equal to this min/init number of executors,
dynamic allocation won't actually request any executors and will just hang
indefinitely with the warning "Initial job has not accepted any resources;
check your cluster UI to ensure that workers are registered and have sufficient
resources".
The cause appears to be that ExecutorAllocationManager does not request any
executors while the application is still initializing, but it still sets the
initial value of numExecutorsTarget to
spark.dynamicAllocation.initialExecutors. Once the job is running and has
submitted its first task, if the first task does not need more than
spark.dynamicAllocation.initialExecutors,
ExecutorAllocationManager.updateAndSyncNumExecutorsTarget() does not think that
it needs to request any executors, so it doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
