[
https://issues.apache.org/jira/browse/SPARK-21840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16208171#comment-16208171
]
Apache Spark commented on SPARK-21840:
--------------------------------------
User 'vanzin' has created a pull request for this issue:
https://github.com/apache/spark/pull/19519
> Allow multiple SparkSubmit invocations in same JVM without polluting system
> properties
> --------------------------------------------------------------------------------------
>
> Key: SPARK-21840
> URL: https://issues.apache.org/jira/browse/SPARK-21840
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core
> Affects Versions: 2.3.0
> Reporter: Marcelo Vanzin
> Priority: Minor
>
> Filing this as a sub-task of SPARK-11035; this feature was discussed as part
> of the PR currently attached to that bug.
> Basically, to allow the launcher library to run applications in-process, the
> easiest way is for it to run the {{SparkSubmit}} class. But that class
> currently propagates configuration to applications by modifying system
> properties.
> That means that when launching multiple applications in that manner in the
> same JVM, the configuration of the first application may leak into the second
> application (or to any other invocation of `new SparkConf()` for that matter).
> This feature is about breaking out the fix for this particular issue from the
> PR linked to SPARK-11035. With the changes in SPARK-21728, the implementation
> can even be further enhanced by providing an actual {{SparkConf}} instance to
> the application, instead of opaque maps.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
