Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5428#discussion_r28841661
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala ---
@@ -621,19 +636,59 @@ object StreamingContext extends Logging {
hadoopConf: Configuration = new Configuration(),
createOnError: Boolean = false
): StreamingContext = {
- val checkpointOption = try {
- CheckpointReader.read(checkpointPath, new SparkConf(), hadoopConf)
- } catch {
- case e: Exception =>
- if (createOnError) {
- None
- } else {
- throw e
- }
- }
+ val checkpointOption = CheckpointReader.read(
+ checkpointPath, new SparkConf(), hadoopConf, createOnError)
checkpointOption.map(new StreamingContext(null, _,
null)).getOrElse(creatingFunc())
}
+
+ /**
+ * Either recreate a StreamingContext from checkpoint data or create a
new StreamingContext.
+ * If checkpoint data exists in the provided `checkpointPath`, then
StreamingContext will be
+ * recreated from the checkpoint data. If the data does not exist, then
the StreamingContext
+ * will be created by called the provided `creatingFunc` on the provided
`sparkContext`. Note
+ * that the SparkConf configuration in the checkpoint data will not be
restored as the
+ * SparkContext has already been created.
+ *
+ * @param checkpointPath Checkpoint directory used in an earlier
StreamingContext program
+ * @param creatingFunc Function to create a new StreamingContext using
the given SparkContext
+ * @param sparkContext SparkContext using which the StreamingContext
will be created
+ */
+ def getOrCreate(
--- End diff --
Do you need this particular overloaded method? Couldn't you just provide
`false` as a default argument to `createOnError` in the next line?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
