[jira] [Updated] (SPARK-10790) Dynamic Allocation does not request any executors if first stage needs less than or equal to spark.dynamicAllocation.initialExecutors
[ https://issues.apache.org/jira/browse/SPARK-10790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-10790: -- Affects Version/s: (was: 1.5.1) 1.5.0 > Dynamic Allocation does not request any executors if first stage needs less > than or equal to spark.dynamicAllocation.initialExecutors > - > > Key: SPARK-10790 > URL: https://issues.apache.org/jira/browse/SPARK-10790 > Project: Spark > Issue Type: Bug > Components: Scheduler >Affects Versions: 1.5.0 >Reporter: Jonathan Kelly >Assignee: Saisai Shao > Fix For: 1.5.2, 1.6.0 > > > If you set spark.dynamicAllocation.initialExecutors > 0 (or > spark.dynamicAllocation.minExecutors, since > spark.dynamicAllocation.initialExecutors defaults to > spark.dynamicAllocation.minExecutors), and the number of tasks in the first > stage of your job is less than or equal to this min/init number of executors, > dynamic allocation won't actually request any executors and will just hang > indefinitely with the warning "Initial job has not accepted any resources; > check your cluster UI to ensure that workers are registered and have > sufficient resources". > The cause appears to be that ExecutorAllocationManager does not request any > executors while the application is still initializing, but it still sets the > initial value of numExecutorsTarget to > spark.dynamicAllocation.initialExecutors. Once the job is running and has > submitted its first task, if the first task does not need more than > spark.dynamicAllocation.initialExecutors, > ExecutorAllocationManager.updateAndSyncNumExecutorsTarget() does not think > that it needs to request any executors, so it doesn't. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-10790) Dynamic Allocation does not request any executors if first stage needs less than or equal to spark.dynamicAllocation.initialExecutors
[ https://issues.apache.org/jira/browse/SPARK-10790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-10790: -- Affects Version/s: (was: 1.5.0) 1.5.1 > Dynamic Allocation does not request any executors if first stage needs less > than or equal to spark.dynamicAllocation.initialExecutors > - > > Key: SPARK-10790 > URL: https://issues.apache.org/jira/browse/SPARK-10790 > Project: Spark > Issue Type: Bug > Components: Scheduler >Affects Versions: 1.5.0 >Reporter: Jonathan Kelly >Assignee: Saisai Shao > Fix For: 1.5.2, 1.6.0 > > > If you set spark.dynamicAllocation.initialExecutors > 0 (or > spark.dynamicAllocation.minExecutors, since > spark.dynamicAllocation.initialExecutors defaults to > spark.dynamicAllocation.minExecutors), and the number of tasks in the first > stage of your job is less than or equal to this min/init number of executors, > dynamic allocation won't actually request any executors and will just hang > indefinitely with the warning "Initial job has not accepted any resources; > check your cluster UI to ensure that workers are registered and have > sufficient resources". > The cause appears to be that ExecutorAllocationManager does not request any > executors while the application is still initializing, but it still sets the > initial value of numExecutorsTarget to > spark.dynamicAllocation.initialExecutors. Once the job is running and has > submitted its first task, if the first task does not need more than > spark.dynamicAllocation.initialExecutors, > ExecutorAllocationManager.updateAndSyncNumExecutorsTarget() does not think > that it needs to request any executors, so it doesn't. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-10790) Dynamic Allocation does not request any executors if first stage needs less than or equal to spark.dynamicAllocation.initialExecutors
[ https://issues.apache.org/jira/browse/SPARK-10790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-10790: -- Priority: Major (was: Critical) Yeah if so then ignore most of this since I thought you were on 1.4.1. I realize you say 1.5.0 here. [~jerryshao] what do you think of this? It looks like there's a missed step in here; when the stage is submitted and it's then allowed to request executors, it never does actually request the initial set of executors? whereas before it happened to have already done that at the outset? Of course, you probably want to set your initial number of executors lower anyway, and that's a workaround, but it shouldn't work this way. > Dynamic Allocation does not request any executors if first stage needs less > than or equal to spark.dynamicAllocation.initialExecutors > - > > Key: SPARK-10790 > URL: https://issues.apache.org/jira/browse/SPARK-10790 > Project: Spark > Issue Type: Bug > Components: Scheduler >Affects Versions: 1.5.0 >Reporter: Jonathan Kelly > > If you set spark.dynamicAllocation.initialExecutors > 0 (or > spark.dynamicAllocation.minExecutors, since > spark.dynamicAllocation.initialExecutors defaults to > spark.dynamicAllocation.minExecutors), and the number of tasks in the first > stage of your job is less than or equal to this min/init number of executors, > dynamic allocation won't actually request any executors and will just hang > indefinitely with the warning "Initial job has not accepted any resources; > check your cluster UI to ensure that workers are registered and have > sufficient resources". > The cause appears to be that ExecutorAllocationManager does not request any > executors while the application is still initializing, but it still sets the > initial value of numExecutorsTarget to > spark.dynamicAllocation.initialExecutors. Once the job is running and has > submitted its first task, if the first task does not need more than > spark.dynamicAllocation.initialExecutors, > ExecutorAllocationManager.updateAndSyncNumExecutorsTarget() does not think > that it needs to request any executors, so it doesn't. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
