Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/7279#discussion_r36038718
--- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala ---
@@ -1448,33 +1468,92 @@ abstract class RDD[T: ClassTag](
/**
* Mark this RDD for checkpointing. It will be saved to a file inside
the checkpoint
- * directory set with SparkContext.setCheckpointDir() and all references
to its parent
+ * directory set with `SparkContext#setCheckpointDir` and all references
to its parent
* RDDs will be removed. This function must be called before any job has
been
* executed on this RDD. It is strongly recommended that this RDD is
persisted in
* memory, otherwise saving it on a file will require recomputation.
*/
- def checkpoint(): Unit = {
+ def checkpoint(): Unit = RDDCheckpointData.synchronized {
+ // NOTE: we use a global lock here due to complexities downstream with
ensuring
+ // children RDD partitions point to the correct parent partitions. In
the future
+ // we should revisit this consideration.
if (context.checkpointDir.isEmpty) {
throw new SparkException("Checkpoint directory has not been set in
the SparkContext")
} else if (checkpointData.isEmpty) {
- // NOTE: we use a global lock here due to complexities downstream
with ensuring
- // children RDD partitions point to the correct parent partitions.
In the future
- // we should revisit this consideration.
- RDDCheckpointData.synchronized {
- checkpointData = Some(new RDDCheckpointData(this))
- }
+ checkpointData = Some(new ReliableRDDCheckpointData(this))
+ }
+ }
--- End diff --
What happens when checkpointData is not empty? If it is not
`ReliableRDDCheckpointData` already, the user is probably doing something wrong
as he/she has called checkpoint() after calling localCheckpoint(). I think
instead of being a complete no-op, we should issue a warning. Otherwise the
user is going to think "I called checkpoint(), why are thinking failing"...
super hard to debug.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
