[
https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173403#comment-14173403
]
Praveen Seluka commented on SPARK-3174:
---------------------------------------
Thanks [~vanzin] for taking a look. Regarding your questions,
- You are right, Doc is not clear on this and just mentions about cached data.
Downscale the executor only if it does not have any cached/shuffle data - that
was the initial plan. Later, If the shuffle files are not part of executors,
then we just need to look at the cached data alone.
- We can actually have a public API if we make it work in both mesos and yarn
modes. These API can be a No-Operation for Standalone mode - is there a problem
with this ?
I just took a brief look at Andrew's PR. Adding these in
TaskSchedulerImpl/TaskSetManager can make it not pluggable and very tied to
that code. For instance, if am looking at changing the dynamic scaling
algorithm at a later point, there is no one place to look at. The proposal I
have, you could just replace the autoscaler class with a totally new one.
> Provide elastic scaling within a Spark application
> --------------------------------------------------
>
> Key: SPARK-3174
> URL: https://issues.apache.org/jira/browse/SPARK-3174
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, YARN
> Affects Versions: 1.0.2
> Reporter: Sandy Ryza
> Assignee: Andrew Or
> Attachments: SPARK-3174design.pdf, SparkElasticScalingDesignB.pdf,
> dynamic-scaling-executors-10-6-14.pdf
>
>
> A common complaint with Spark in a multi-tenant environment is that
> applications have a fixed allocation that doesn't grow and shrink with their
> resource needs. We're blocked on YARN-1197 for dynamically changing the
> resources within executors, but we can still allocate and discard whole
> executors.
> It would be useful to have some heuristics that
> * Request more executors when many pending tasks are building up
> * Discard executors when they are idle
> See the latest design doc for more information.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
