Marcelo Vanzin created SPARK-8055:
-------------------------------------
Summary: Spark Launcher Improvements
Key: SPARK-8055
URL: https://issues.apache.org/jira/browse/SPARK-8055
Project: Spark
Issue Type: Umbrella
Components: Spark Core
Affects Versions: 1.4.0
Reporter: Marcelo Vanzin
Filing a bug to track different enhancements to the Spark launcher library.
Please file sub-tasks for each particular enhancement instead of tagging PRs
with this bug's number.
After some discussion in the mailing list, people have requested different
enhancements to the library. I'll try to capture those here but feel free to
add more in the comments.
- Missing information about the launched Spark application.
Currently the library returns an opaque "Process" object that doesn't have a
lot of Spark-related functionality. It would be useful to get at least some
information about the underlying process; in the very least the application ID
of the actual Spark job. Other useful information could be, for example, the
current status of the submitted job.
- Ability to control the underlying application.
The "Process" object only allows you to kill the underlying application. It
would be better to have better application-level APIs to try to stop the
application more cleanly (e.g. by asking the cluster manager to kill it, or by
stopping the SparkContext in client mode).
- Ability to run Spark applications in the same JVM.
This could potentially be done today for cluster mode apps without getting bit
by the limitations of SparkContext. In the long run, it would be nice to also
support client mode apps.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
