Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/7279#discussion_r35393329
--- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala ---
@@ -1459,20 +1479,82 @@ abstract class RDD[T: ClassTag](
// children RDD partitions point to the correct parent partitions.
In the future
// we should revisit this consideration.
RDDCheckpointData.synchronized {
- checkpointData = Some(new RDDCheckpointData(this))
+ checkpointData = Some(new ReliableRDDCheckpointData(this))
}
}
}
/**
- * Return whether this RDD has been checkpointed or not
+ * Mark this RDD for local checkpointing using Spark's existing caching
layer.
+ *
+ * This method is for users who wish to truncate RDD lineages while
skipping the expensive
+ * step of replicating the materialized data in a reliable distributed
file system. This is
+ * useful for RDDs with long lineages that need to be truncated
periodically (e.g. GraphX).
+ *
+ * Local checkpointing sacrifices fault-tolerance for performance. In
particular,
+ * checkpointed data is written to ephemeral local storage (memory or
disk) instead of
--- End diff --
instead of ~to~ a reliable, ~distributed~ fault-tolerant storage like HDFS.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
