Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/7279#discussion_r35393327
--- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala ---
@@ -1459,20 +1479,82 @@ abstract class RDD[T: ClassTag](
// children RDD partitions point to the correct parent partitions.
In the future
// we should revisit this consideration.
RDDCheckpointData.synchronized {
- checkpointData = Some(new RDDCheckpointData(this))
+ checkpointData = Some(new ReliableRDDCheckpointData(this))
}
}
}
/**
- * Return whether this RDD has been checkpointed or not
+ * Mark this RDD for local checkpointing using Spark's existing caching
layer.
+ *
+ * This method is for users who wish to truncate RDD lineages while
skipping the expensive
+ * step of replicating the materialized data in a reliable distributed
file system. This is
+ * useful for RDDs with long lineages that need to be truncated
periodically (e.g. GraphX).
+ *
+ * Local checkpointing sacrifices fault-tolerance for performance. In
particular,
+ * checkpointed data is written to ephemeral local storage (memory or
disk) instead of
+ * to a reliable distributed storage. The effect is that if an executor
fails during the
+ * computation, the checkpointed data may no longer be accessible,
causing an irrecoverable
+ * job failure.
+ *
+ * This is NOT safe to use with dynamic allocation, which removes
executors along
+ * with their cached blocks. If you must use both features, you are
advised to set
+ * `spark.dynamicAllocation.cachedExecutorIdleTimeout` to a high value.
+ *
+ * The checkpoint directory set through `SparkContext#setCheckpointDir`
is not read.
--- End diff --
is not used.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
