[ https://issues.apache.org/jira/browse/SPARK-21840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16208171#comment-16208171 ]
Apache Spark commented on SPARK-21840: -------------------------------------- User 'vanzin' has created a pull request for this issue: https://github.com/apache/spark/pull/19519 > Allow multiple SparkSubmit invocations in same JVM without polluting system > properties > -------------------------------------------------------------------------------------- > > Key: SPARK-21840 > URL: https://issues.apache.org/jira/browse/SPARK-21840 > Project: Spark > Issue Type: New Feature > Components: Spark Core > Affects Versions: 2.3.0 > Reporter: Marcelo Vanzin > Priority: Minor > > Filing this as a sub-task of SPARK-11035; this feature was discussed as part > of the PR currently attached to that bug. > Basically, to allow the launcher library to run applications in-process, the > easiest way is for it to run the {{SparkSubmit}} class. But that class > currently propagates configuration to applications by modifying system > properties. > That means that when launching multiple applications in that manner in the > same JVM, the configuration of the first application may leak into the second > application (or to any other invocation of `new SparkConf()` for that matter). > This feature is about breaking out the fix for this particular issue from the > PR linked to SPARK-11035. With the changes in SPARK-21728, the implementation > can even be further enhanced by providing an actual {{SparkConf}} instance to > the application, instead of opaque maps. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org