[
https://issues.apache.org/jira/browse/SPARK-7699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14560512#comment-14560512
]
Saisai Shao commented on SPARK-7699:
------------------------------------
I think timeout is a necessary and simple way to handle this problem. Beside
this fixed timeout, we could also consider whether current number of executor
is larger than the initial target number, if satisfied, we could start the
scheduler (no need to wait until the timeout) to further ramp down or increase
the executor number according to the load, this will keep the initial started
executor to keep alive for at least 60 seconds, and to avoid unnecessarily
ramping down in-case user job is later than the scheduler task to update the
executor number.
Besides I think "avoid adjusting targetNumExecutors down before the first job
is submitted" is another possible solution, but if user who started spark-shell
but has no further action for a long time, this will keep the unnecessary
executors retain for a long time.
I have a patch according to timeout mechanism, please help to review, any
comment is greatly appreciated.
> Number of executors can be reduced from initial before work is scheduled
> ------------------------------------------------------------------------
>
> Key: SPARK-7699
> URL: https://issues.apache.org/jira/browse/SPARK-7699
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Reporter: meiyoula
> Priority: Minor
>
> spark.dynamicAllocation.minExecutors 2
> spark.dynamicAllocation.initialExecutors 3
> spark.dynamicAllocation.maxExecutors 4
> Just run the spark-shell with above configurations, the initial executor
> number is 2.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
