[
https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14125235#comment-14125235
]
Sandy Ryza commented on SPARK-3174:
-----------------------------------
To be clear, by YARN shuffle you mean the MR2 approach where shuffle data is
served by an auxiliary service living in the NodeManager? I think that could
definitely be beneficial. Though it does have its drawbacks down the line,
like more difficulty in accounting for and throttling disk and network IO.
For Hive-on-Spark's needs, the main motivation is that someone who leaves their
Hive session open but idle shouldn't be holding on to a bunch of cluster
resources. So, for this purpose, it might be sufficient to only discard
executors when no jobs are running. In that case, we wouldn't need to worry
about shuffle data at all.
Also, do you know when shuffle data gets deleted? After the stage that's
fetching it completes or after the job completes?
> Under YARN, add and remove executors based on load
> --------------------------------------------------
>
> Key: SPARK-3174
> URL: https://issues.apache.org/jira/browse/SPARK-3174
> Project: Spark
> Issue Type: Improvement
> Components: YARN
> Affects Versions: 1.0.2
> Reporter: Sandy Ryza
> Assignee: Andrew Or
> Attachments: SPARK-3174design.pdf
>
>
> A common complaint with Spark in a multi-tenant environment is that
> applications have a fixed allocation that doesn't grow and shrink with their
> resource needs. We're blocked on YARN-1197 for dynamically changing the
> resources within executors, but we can still allocate and discard whole
> executors.
> I think it would be useful to have some heuristics that
> * Request more executors when many pending tasks are building up
> * Request more executors when RDDs can't fit in memory
> * Discard executors when few tasks are running / pending and there's not much
> in memory
> Bonus points: migrate blocks from executors we're about to discard to
> executors with free space.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
