Marcelo Vanzin created SPARK-21840:
--------------------------------------
Summary: Allow multiple SparkSubmit invocations in same JVM
without polluting system properties
Key: SPARK-21840
URL: https://issues.apache.org/jira/browse/SPARK-21840
Project: Spark
Issue Type: New Feature
Components: Spark Core
Affects Versions: 2.3.0
Reporter: Marcelo Vanzin
Priority: Minor
Filing this as a sub-task of SPARK-11035; this feature was discussed as part of
the PR currently attached to that bug.
Basically, to allow the launcher library to run applications in-process, the
easiest way is for it to run the {{SparkSubmit}} class. But that class
currently propagates configuration to applications by modifying system
properties.
That means that when launching multiple applications in that manner in the same
JVM, the configuration of the first application may leak into the second
application (or to any other invocation of `new SparkConf()` for that matter).
This feature is about breaking out the fix for this particular issue from the
PR linked to SPARK-11035. With the changes in SPARK-21728, the implementation
can even be further enhanced by providing an actual {{SparkConf}} instance to
the application, instead of opaque maps.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
