[
https://issues.apache.org/jira/browse/SPARK-4194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14193732#comment-14193732
]
Josh Rosen commented on SPARK-4194:
-----------------------------------
I've marked this a blocker of SPARK-4180, an issue that tries to add exceptions
when users try to create multiple active SparkContexts in the same JVM.
PySpark already guards against this, but earlier versions of its error-checking
code ran into issues where users would fail their initial attempt to create a
SparkContext and then be unable to create new ones because we didn't clear the
{{activeSparkContext}} variable after the constructor threw an exception.
To fix this, we need to wrap the constructor code in a {{try}} block.
> Exceptions thrown during SparkContext or SparkEnv construction might lead to
> resource leaks or corrupted global state
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-4194
> URL: https://issues.apache.org/jira/browse/SPARK-4194
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Reporter: Josh Rosen
> Priority: Critical
>
> The SparkContext and SparkEnv constructors instantiate a bunch of objects
> that may need to be cleaned up after they're no longer needed. If an
> exception is thrown during SparkContext or SparkEnv construction (e.g. due to
> a bad configuration setting), then objects created earlier in the constructor
> may not be properly cleaned up.
> This is unlikely to cause problems for batch jobs submitted through
> {{spark-submit}}, since failure to construct SparkContext will probably cause
> the JVM to exit, but it is a potentially serious issue in interactive
> environments where a user might attempt to create SparkContext with some
> configuration, fail due to an error, and re-attempt the creation with new
> settings. In this case, resources from the previous creation attempt might
> not have been cleaned up and could lead to confusing errors (especially if
> the old, leaked resources share global state with the new SparkContext).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
