[ https://issues.apache.org/jira/browse/SPARK-8055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14570000#comment-14570000 ]
Marcelo Vanzin commented on SPARK-8055: --------------------------------------- /cc [~klmarkey] [~chester.c...@webwarecorp.com] > Spark Launcher Improvements > --------------------------- > > Key: SPARK-8055 > URL: https://issues.apache.org/jira/browse/SPARK-8055 > Project: Spark > Issue Type: Umbrella > Components: Spark Core > Affects Versions: 1.4.0 > Reporter: Marcelo Vanzin > > Filing a bug to track different enhancements to the Spark launcher library. > Please file sub-tasks for each particular enhancement instead of tagging PRs > with this bug's number. > After some discussion in the mailing list, people have requested different > enhancements to the library. I'll try to capture those here but feel free to > add more in the comments. > - Missing information about the launched Spark application. > Currently the library returns an opaque "Process" object that doesn't have a > lot of Spark-related functionality. It would be useful to get at least some > information about the underlying process; in the very least the application > ID of the actual Spark job. Other useful information could be, for example, > the current status of the submitted job. > - Ability to control the underlying application. > The "Process" object only allows you to kill the underlying application. It > would be better to have better application-level APIs to try to stop the > application more cleanly (e.g. by asking the cluster manager to kill it, or > by stopping the SparkContext in client mode). > - Ability to run Spark applications in the same JVM. > This could potentially be done today for cluster mode apps without getting > bit by the limitations of SparkContext. In the long run, it would be nice to > also support client mode apps. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org