[
https://issues.apache.org/jira/browse/SPARK-16158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15346744#comment-15346744
]
Nezih Yigitbasi commented on SPARK-16158:
-----------------------------------------
Thanks [~sowen] for your input, I understand your concern. Our end goal is to
experiment with different heuristics and provide some of these out of the box
where users can pick any of them depending on their workload, what they want to
optimize, etc. So this is really the first step of this investigation. The main
reason we want to explore new heuristics is some shortcomings of the default
heuristic that we noticed with some of our jobs. For example, if a job has
short tasks (say a few hundred ms) the exponential ramp up logic results in a
large number of executors staying idle (by the time containers are allocated
the tasks were done). Another shortcoming we noticed is at stage boundaries if
there is a straggler, which is not uncommon for complex production jobs that we
have, the default heuristic kills all the executors as most of them are idle,
and when the stage is done it takes some time to ramp up again to a decent
capacity (so the ramp down/decay process should be more "gentle"). I wonder
what other users/committers think too. Especially if other users can share
their production experience with the default dynamic allocation heuristic that
would be super helpful for this discussion.
> Support pluggable dynamic allocation heuristics
> -----------------------------------------------
>
> Key: SPARK-16158
> URL: https://issues.apache.org/jira/browse/SPARK-16158
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Reporter: Nezih Yigitbasi
>
> It would be nice if Spark supports plugging in custom dynamic allocation
> heuristics. This feature would be useful for experimenting with new
> heuristics and also useful for plugging in different heuristics per job etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
