GitHub user JoshRosen reopened a pull request:
https://github.com/apache/spark/pull/3121
[SPARK-4180] [Core] Prevent creation of multiple active SparkContexts
This patch adds error-detection logic to throw an exception when attempting
to create multiple active SparkContexts in the same JVM, since this is
currently unsupported and has been known to cause confusing behavior (see
SPARK-2243 for more details).
**The solution implemented here is only a partial fix.** A complete fix
would have the following properties:
1. Only one SparkContext may ever be under construction at any given time.
2. Once a SparkContext has been successfully constructed, any subsequent
construction attempts should fail until the active SparkContext is stopped.
3. If the SparkContext constructor throws an exception, then all resources
created in the constructor should be cleaned up (SPARK-4194).
4. If a user attempts to create a SparkContext but the creation fails, then
the user should be able to create new SparkContexts.
This PR only provides 1) and 2) (and partial support for 4) in two limited
cases); we should be able to provide all of these properties, but the correct
fix will involve larger changes to SparkContext's construction /
initialization, so we'll target it for a different Spark release.
### The correct solution:
I think that the correct way to do this would be to move the construction
of SparkContext's dependencies into a static method in the SparkContext
companion object. Specifically, we could make the default SparkContext
constructor `private` and change it to accept a `SparkContextDependencies`
object that contains all of SparkContext's dependencies (e.g. DAGScheduler,
ContextCleaner, etc.). Secondary constructors could call a method on the
SparkContext companion object to create the `SparkContextDependencies` and pass
the result to the primary SparkContext constructor. For example:
```scala
class SparkContext private (deps: SparkContextDependencies) {
def this(conf: SparkConf) {
this(SparkContext.getDeps(conf))
}
}
object SparkContext(
private[spark] def getDeps(conf: SparkConf): SparkContextDependencies =
synchronized {
if (anotherSparkContextIsActive) { throw Exception(...) }
var dagScheduler: DAGScheduler = null
try {
dagScheduler = new DAGScheduler(...)
[...]
} catch {
case e: Exception =>
Option(dagScheduler).foreach(_.stop())
[...]
}
SparkContextDependencies(dagScheduler, ....)
}
}
```
This gives us mutual exclusion and ensures that any resources created
during the failed SparkContext initialization are properly cleaned up.
This indirection is necessary to maintain binary compatibility. In
retrospect, it would have been nice if SparkContext had no private constructors
and could only be created through builder / factory methods on its companion
object, since this buys us lots of flexibility and makes dependency injection
easier.
### Alternative solutions:
As an alternative solution, we could refactor SparkContext's primary
constructor to perform all object creation in a giant `try-finally` block.
Unfortunately, this will require us to turn a bunch of `vals` into `vars` so
that they can be assigned from the `try` block. If we still want `vals`, we
could wrap each `val` in its own `try` block (since the try block can return a
value), but this will lead to extremely messy code and won't guard against the
introduction of future code which doesn't properly handle failures.
The more complex approach outlined above gives us some nice dependency
injection benefits, so I think that might be preferable to a `var`-ification.
### The limitations of this PR's solution:
This PR ensures that _an error_ will occur if multiple SparkContexts are
active at the same time, even if multiple threads race to construct those
contexts. The key problem is the fact that we don't hold the
`SparkContext.SPARK_CONTEXT_CONSTRUCTOR_LOCK` for the entire duration of the
constructor, so we don't have a way to distinguish between a SparkContext
creation attempt that failed because another SparkContext was undergoing
construction and a case where a previous construction attempt failed and left
some state that wasn't cleaned up. This means that it's possible for
SparkContext's constructor to throw an exception in a way that prevents any
future creations of SparkContext until the JVM is restarted.
This is unlikely to be a problem in practice, though, especially since I've
added special-casing so that this issue doesn't affect the most common causes
of SparkContext construction errors (e.g. forgetting to set the master URL or
application name). In case it is, I've added an undocumented
`spark.driver.disableMultipleSparkContextsErrorChecking` option which turns
this error into a warning.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark SPARK-4180
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3121.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3121
----
commit afaa7e37cce76da8173ff556d217a67a1633426a
Author: Josh Rosen <[email protected]>
Date: 2014-11-05T23:06:13Z
[SPARK-4180] Prevent creations of multiple active SparkContexts.
commit 918e8782eb94ea59330f285cbbaaa7aaa7235583
Author: Josh Rosen <[email protected]>
Date: 2014-11-05T23:50:44Z
Document "one SparkContext per JVM" limitation.
commit c4d35a255c5b912f83df8b55557edb1f557fa251
Author: Josh Rosen <[email protected]>
Date: 2014-11-06T02:22:30Z
Log long form of creation site to aid debugging.
commit d0437ebbabf623f3e102ffaae166baae99786d90
Author: Josh Rosen <[email protected]>
Date: 2014-11-06T02:23:22Z
StreamingContext.stop() should stop SparkContext even if StreamingContext
has not been started yet.
commit 06c5c541cf8fcd72715fd304ea8f12b7cfbc8463
Author: Josh Rosen <[email protected]>
Date: 2014-11-06T02:24:15Z
Add / improve SparkContext cleanup in streaming BasicOperationsSuite
commit 1c660708fe2e720d0eed69387aeebad0b2a964e2
Author: Josh Rosen <[email protected]>
Date: 2014-11-06T04:53:09Z
Merge remote-tracking branch 'origin/master' into SPARK-4180
commit ed17e14621f86137ccdc5408c9d10b3e683903f6
Author: Josh Rosen <[email protected]>
Date: 2014-11-07T01:00:18Z
Address review feedback; expose hack workaround for existing unit tests.
commit 4629d5c3b9305353526992760a80e5ff0d5ae9aa
Author: Josh Rosen <[email protected]>
Date: 2014-11-07T01:02:24Z
Set spark.driver.allowMultipleContexts=true in tests.
commit 7ba6db8e89a18cfa6e56c8b390998a2abb7b8560
Author: Josh Rosen <[email protected]>
Date: 2014-11-07T01:09:28Z
Add utility to set system properties in tests.
commit a1cba6511a2d1c60364c300bf89b29a38f78a350
Author: Josh Rosen <[email protected]>
Date: 2014-11-10T08:08:46Z
Merge remote-tracking branch 'origin/master' into SPARK-4180
Conflicts:
streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala
commit 79a7e6ffa7d450246d04816fd493278a8bc60bc5
Author: Josh Rosen <[email protected]>
Date: 2014-11-10T08:10:34Z
Fix commented out test
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
