Github user the-sea commented on a diff in the pull request:
https://github.com/apache/spark/pull/12273#discussion_r59125550
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -2192,8 +2192,8 @@ object SparkContext extends Logging {
sc: SparkContext,
allowMultipleContexts: Boolean): Unit = {
SPARK_CONTEXT_CONSTRUCTOR_LOCK.synchronized {
- assertNoOtherContextIsRunning(sc, allowMultipleContexts)
contextBeingConstructed = Some(sc)
+ assertNoOtherContextIsRunning(sc, allowMultipleContexts)
--- End diff --
when a SparkContext object is created, the ```contextBeingConstructed ```
will be set ```None``` in ```SparkContext.setActiveContext``` method, so when
we create a SparkContext object again,the
```SparkContext.markPartiallyConstructed``` method will not throw any
exceptions,because ```contextBeingConstructed``` is ```None``` right now,
```contextBeingConstructed.foreach``` method body will not be executed in ```
assertNoOtherContextIsRunning``` method.
```scala
private def assertNoOtherContextIsRunning(
sc: SparkContext,
allowMultipleContexts: Boolean): Unit = {
SPARK_CONTEXT_CONSTRUCTOR_LOCK.synchronized {
contextBeingConstructed.foreach { otherContext =>
if (otherContext ne sc) { // checks for reference equality
// Since otherContext might point to a partially-constructed
context, guard against
// its creationSite field being null:
val otherContextCreationSite =
Option(otherContext.creationSite).map(_.longForm).getOrElse("unknown location")
val warnMsg = "Another SparkContext is being constructed (or
threw an exception in its" +
" constructor). This may indicate an error, since only one
SparkContext may be" +
" running in this JVM (see SPARK-2243)." +
s" The other SparkContext was created
at:\n$otherContextCreationSite"
logWarning(warnMsg)
}
if (activeContext.get() != null) {
val ctx = activeContext.get()
val errMsg = "Only one SparkContext may be running in this JVM
(see SPARK-2243)." +
" To ignore this error, set spark.driver.allowMultipleContexts
= true. " +
s"The currently running SparkContext was created
at:\n${ctx.creationSite.longForm}"
val exception = new SparkException(errMsg)
if (allowMultipleContexts) {
logWarning("Multiple running SparkContexts detected in the same
JVM!", exception)
} else {
throw exception
}
}
}
}
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
