[
https://issues.apache.org/jira/browse/SPARK-11459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14990676#comment-14990676
]
Ryan Williams commented on SPARK-11459:
---------------------------------------
I'm mostly interested in saving RDDs to disk with kryo-serde
([SPARK-11461|https://issues.apache.org/jira/browse/SPARK-11461]).
The existing checkpoint APIs are functionally exactly what I want, but they
mandate putting a UUID in the directory name and fixing the basename to the RDD
ID, somewhat unnecessarily.
Letting the user opt in to specifying the path is a simple way to get at the
functionality that I want without having to do something possibly more invasive
e.g. for SPARK-11461, and there's not really a danger of it conflicting with
existing checkpoint usages. It could even be exposed via a different method on
SparkContext/RDD, if overloading the semantics of {{checkpoint}} is the concern.
Another, orthogonal option I've worked on a little is basically copy/pasting a
bunch of the checkpointing logic into a Spark package that hangs methods off of
SparkContext and RDD that do [checkpointing with configurable path naming], but
that's an unideal use-case for Spark packages: it is a bunch of code that's
already in Spark, that I'd have to keep up to date, etc.
> Allow configuring checkpoint dir, filenames
> -------------------------------------------
>
> Key: SPARK-11459
> URL: https://issues.apache.org/jira/browse/SPARK-11459
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 1.5.1
> Reporter: Ryan Williams
> Priority: Minor
>
> I frequently want to persist some RDDs to disk and choose the names of the
> files that they are saved as.
> Currently, the {{RDD.checkpoint}} flow [writes to a directory with a UUID in
> its
> name|https://github.com/apache/spark/blob/v1.5.1/core/src/main/scala/org/apache/spark/SparkContext.scala#L2050],
> and the file is [always named after the RDD's
> ID|https://github.com/apache/spark/blob/v1.5.1/core/src/main/scala/org/apache/spark/rdd/ReliableRDDCheckpointData.scala#L96].
> Is there any reason not to allow the user to e.g. pass a string to
> {{RDD.checkpoint}} that will set the location that the RDD is checkpointed to?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
