[
https://issues.apache.org/jira/browse/SPARK-7699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14558764#comment-14558764
]
Sean Owen commented on SPARK-7699:
----------------------------------
initialExecutors? I think it's there for the ramp-up case really. If load will
start "soon", but your minimum is 1 because load is variable, it's best to not
have to ramp up through 1, 2, 4, 8 executors if you need 100.
The problem is evaluating load before any load has had a chance to schedule.
Ramping down at all is bad if, actually, load is applied right away.
I'd rather not add another lever here, but is it principled to wait for some
multiple of the RM heartbeat here? so that the allocation isn't changed until
the RM has had a fair chance to allocate resources? Sure, bets are off if there
is a delay in scheduling but what can you do? nothing breaks here it's just
suboptimal then.
> Number of executors can be reduced from initial before work is scheduled
> ------------------------------------------------------------------------
>
> Key: SPARK-7699
> URL: https://issues.apache.org/jira/browse/SPARK-7699
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Reporter: meiyoula
> Priority: Minor
>
> spark.dynamicAllocation.minExecutors 2
> spark.dynamicAllocation.initialExecutors 3
> spark.dynamicAllocation.maxExecutors 4
> Just run the spark-shell with above configurations, the initial executor
> number is 2.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
