GitHub user tdas opened a pull request:
https://github.com/apache/spark/pull/5428
[Spark-6752]
Currently if you want to create a StreamingContext from checkpoint
information, the system will create a new SparkContext. This prevent
StreamingContext to be recreated from checkpoints in managed environments where
SparkContext is precreated.
The solution in this PR: Introduce the following methods on StreamingContext
1. `new StreamingContext(checkpointDirectory, sparkContext)`
Recreate StreamingContext from checkpoint using the provided SparkContext
2. `StreamingContext.getOrCreate(checkpointDirectory, sparkContext,
createFunction: SparkContext => StreamingContext)`
If checkpoint file exists, then recreate StreamingContext using the
provided SparkContext (that is, call 1.), else create StreamingContext using
the provided createFunction
TODO: the corresponding Java and Python API has to be added as well.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tdas/spark SPARK-6752
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5428.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5428
----
commit 204814ea2be868257b32f686e1455254f5d60582
Author: Tathagata Das <[email protected]>
Date: 2015-04-08T22:30:48Z
Added StreamingContext.getOrCreate with existing SparkContext
commit 36a782356e9fc032a0d6a42251ae82e2af25aeaa
Author: Tathagata Das <[email protected]>
Date: 2015-04-08T22:32:02Z
Minor changes.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
