Timothy Chen created SPARK-8873:
-----------------------------------
Summary: Support cleaning up shuffle files for drivers launched
with Mesos
Key: SPARK-8873
URL: https://issues.apache.org/jira/browse/SPARK-8873
Project: Spark
Issue Type: Improvement
Reporter: Timothy Chen
With dynamic allocation enabled with Mesos, drivers can launch with shuffle
data cached in the external shuffle service.
However, there is no reliable way to let the shuffle service clean up the
shuffle data when the driver exits, since it may crash before it notifies the
shuffle service and shuffle data will be cached forever.
We need to implement a reliable way to detect driver termination and clean up
shuffle data accordingly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
