Github user ferdonline commented on a diff in the pull request:
https://github.com/apache/spark/pull/19805#discussion_r153722443
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -537,9 +536,55 @@ class Dataset[T] private[sql](
*/
@Experimental
@InterfaceStability.Evolving
- def checkpoint(eager: Boolean): Dataset[T] = {
+ def checkpoint(eager: Boolean = true): Dataset[T] = _checkpoint(eager =
eager)
+
+ /**
+ * Eagerly locally checkpoints a Dataset and return the new Dataset.
Checkpointing can be
+ * used to truncate the logical plan of this Dataset, which is
especially useful in iterative
+ * algorithms where the plan may grow exponentially. Local checkpoints
are written to executor
+ * storage and despite potentially faster they are unreliable and may
compromise job completion.
+ *
+ * @group basic
+ * @since 2.3.0
+ */
+ @Experimental
+ @InterfaceStability.Evolving
+ def localCheckpoint(): Dataset[T] = _checkpoint(eager = true, local =
true)
+
+ /**
+ * Locally checkpoints a Dataset and return the new Dataset.
Checkpointing can be used to truncate
+ * the logical plan of this Dataset, which is especially useful in
iterative algorithms where the
+ * plan may grow exponentially. Local checkpoints are written to
executor storage and despite
+ * potentially faster they are unreliable and may compromise job
completion.
+ *
+ * @group basic
+ * @since 2.3.0
+ */
+ @Experimental
+ @InterfaceStability.Evolving
+ def localCheckpoint(eager: Boolean = true): Dataset[T] =
_checkpoint(eager = eager, local = true)
+
+ /**
+ * Returns a checkpointed version of this Dataset. Checkpointing can be
used to truncate the
+ * logical plan of this Dataset, which is especially useful in iterative
algorithms where the
+ * plan may grow exponentially.
+ * By default reliable checkpoints are created and saved to files inside
the checkpoint
+ * directory set with `SparkContext#setCheckpointDir`. If local is set
to true a local checkpoint
+ * is performed instead. Local checkpoints are written to executor
storage and despite
+ * potentially faster they are unreliable and may compromise job
completion.
+ *
+ * @group basic
+ * @since 2.3.0
+ */
+ @Experimental
+ @InterfaceStability.Evolving
+ private[sql] def _checkpoint(eager: Boolean, local: Boolean = false):
Dataset[T] = {
--- End diff --
I changed according to the first review. I also agree to be private so that
"there's only one way of doing it" when using the API. But if you have a strong
feeling I can surely change
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
