[
https://issues.apache.org/jira/browse/SPARK-8055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14570000#comment-14570000
]
Marcelo Vanzin commented on SPARK-8055:
---------------------------------------
/cc [~klmarkey] [[email protected]]
> Spark Launcher Improvements
> ---------------------------
>
> Key: SPARK-8055
> URL: https://issues.apache.org/jira/browse/SPARK-8055
> Project: Spark
> Issue Type: Umbrella
> Components: Spark Core
> Affects Versions: 1.4.0
> Reporter: Marcelo Vanzin
>
> Filing a bug to track different enhancements to the Spark launcher library.
> Please file sub-tasks for each particular enhancement instead of tagging PRs
> with this bug's number.
> After some discussion in the mailing list, people have requested different
> enhancements to the library. I'll try to capture those here but feel free to
> add more in the comments.
> - Missing information about the launched Spark application.
> Currently the library returns an opaque "Process" object that doesn't have a
> lot of Spark-related functionality. It would be useful to get at least some
> information about the underlying process; in the very least the application
> ID of the actual Spark job. Other useful information could be, for example,
> the current status of the submitted job.
> - Ability to control the underlying application.
> The "Process" object only allows you to kill the underlying application. It
> would be better to have better application-level APIs to try to stop the
> application more cleanly (e.g. by asking the cluster manager to kill it, or
> by stopping the SparkContext in client mode).
> - Ability to run Spark applications in the same JVM.
> This could potentially be done today for cluster mode apps without getting
> bit by the limitations of SparkContext. In the long run, it would be nice to
> also support client mode apps.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
