Patrick Wendell created SPARK-7292:
--------------------------------------
Summary: Provide operator to truncate lineage without persisting
RDD's
Key: SPARK-7292
URL: https://issues.apache.org/jira/browse/SPARK-7292
Project: Spark
Issue Type: New Feature
Components: Spark Core
Reporter: Patrick Wendell
Checkpointing exists in Spark to truncate a lineage chain. I've heard requests
from some users to allow truncation of lineage in a way that is "cheap" and
doesn't serialized and persist the RDD. This is possible if the user is willing
to forgo fault tolerance for that RDD (for instance, for shorter running jobs
or ones that use a small number of machines). It's pretty easy to allow this so
we should look into it for Spark 1.5.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
