[
https://issues.apache.org/jira/browse/SPARK-4280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14200801#comment-14200801
]
Sandy Ryza commented on SPARK-4280:
-----------------------------------
My thinking was that it would just be based on whether the node reports it's
storing blocks. So, e.g., applications wouldn't need to explicitly uncache the
RDDs in situations where Spark garbage collects them because their references
go away.
Ideally we would make a special case for broadcast variables, because they're
not unique to any node. Will look into whether there's a good way to do this.
> In dynamic allocation, add option to never kill executors with cached blocks
> ----------------------------------------------------------------------------
>
> Key: SPARK-4280
> URL: https://issues.apache.org/jira/browse/SPARK-4280
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 1.2.0
> Reporter: Sandy Ryza
>
> Even with the external shuffle service, this is useful in situations like
> Hive on Spark where a query might require caching some data. We want to be
> able to give back executors after the job ends, but not during the job if it
> would delete intermediate results.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
