GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/3160
[SPARK-4301] StreamingContext should not allow start() to be called after
calling stop()
In Spark 1.0.0+, calling `stop()` on a StreamingContext that has not been
started is a no-op which has no side-effects. This allows users to call
`stop()` on a fresh StreamingContext followed by `start()`. I believe that this
almost always indicates an error and is not behavior that we should support.
Since we don't allow `start() stop() start()` then I don't think it makes sense
to allow `stop() start()`.
The current behavior can lead to resource leaks when StreamingContext
constructs its own SparkContext: if I call `stop(stopSparkContext=True)`, then
I expect StreamingContext's underlying SparkContext to be stopped irrespective
of whether the StreamingContext has been started. This is useful when writing
unit test fixtures.
Prior discussions:
- https://github.com/apache/spark/pull/3053#discussion-diff-19710333R490
- https://github.com/apache/spark/pull/3121#issuecomment-61927353
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark SPARK-4301
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3160.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3160
----
commit 5558e703b53fc13e54280e0bb92b3db6eea664f3
Author: Josh Rosen <[email protected]>
Date: 2014-11-06T02:23:22Z
StreamingContext.stop() should stop SparkContext even if StreamingContext
has not been started yet.
commit 813e4717472b585dbb4b92795ccca96d374679d9
Author: Josh Rosen <[email protected]>
Date: 2014-11-07T18:54:55Z
Revert workaround added in
https://github.com/apache/spark/pull/3053/files#diff-e144dbee130ed84f9465853ddce65f8eR49
commit 5142517536ed59eda8fed6609fe263e95c94ad0e
Author: Josh Rosen <[email protected]>
Date: 2014-11-07T19:17:29Z
Add tests; improve Scaladoc.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
