Ryan Williams created SPARK-9197:
------------------------------------
Summary: Cached RDD partitions are lost when executors are
dynamically deallocated
Key: SPARK-9197
URL: https://issues.apache.org/jira/browse/SPARK-9197
Project: Spark
Issue Type: Bug
Components: YARN
Affects Versions: 1.4.1
Reporter: Ryan Williams
Currently, dynamic allocation cleans up executors that have not run any tasks
for a certain amount of time.
However, this often leads to cached RDD partitions being lost.
Should dynamic allocation leave executors alone that have cached partitions?
Should this be configurable?
Is there any interest in code that would shuffle cached partitions around in
preparation for executor-deallocation, to avoid this? Such logic could be
useful in general for maintaining persisted RDDs across executor churn.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
