Repository: spark Updated Branches: refs/heads/master f039d964d -> 4a426ff8a
[SPARK-17437] Add uiWebUrl to JavaSparkContext and pyspark.SparkContext ## What changes were proposed in this pull request? The Scala version of `SparkContext` has a handy field called `uiWebUrl` that tells you which URL the SparkUI spawned by that instance lives at. This is often very useful because the value for `spark.ui.port` in the config is only a suggestion; if that port number is taken by another Spark instance on the same machine, Spark will just keep incrementing the port until it finds a free one. So, on a machine with a lot of running PySpark instances, you often have to start trying all of them one-by-one until you find your application name. Scala users have a way around this with `uiWebUrl` but Java and Python users do not. This pull request fixes this in the most straightforward way possible, simply propagating this field through the `JavaSparkContext` and into pyspark through the Java gateway. Please let me know if any additional documentation/testing is needed. ## How was this patch tested? Existing tests were run to make sure there were no regressions, and a binary distribution was created and tested manually for the correct value of `sc.uiWebPort` in a variety of circumstances. Author: Adrian Petrescu <apetr...@gmail.com> Closes #15000 from apetresc/pyspark-uiweburl. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/4a426ff8 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/4a426ff8 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/4a426ff8 Branch: refs/heads/master Commit: 4a426ff8aea4faa31a3016a453dec5b7954578dd Parents: f039d96 Author: Adrian Petrescu <apetr...@gmail.com> Authored: Tue Sep 20 10:49:02 2016 +0100 Committer: Sean Owen <so...@cloudera.com> Committed: Tue Sep 20 10:49:02 2016 +0100 ---------------------------------------------------------------------- python/pyspark/context.py | 5 +++++ 1 file changed, 5 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/4a426ff8/python/pyspark/context.py ---------------------------------------------------------------------- diff --git a/python/pyspark/context.py b/python/pyspark/context.py index 2744bb9..5c32f8e 100644 --- a/python/pyspark/context.py +++ b/python/pyspark/context.py @@ -333,6 +333,11 @@ class SparkContext(object): return self._jsc.sc().applicationId() @property + def uiWebUrl(self): + """Return the URL of the SparkUI instance started by this SparkContext""" + return self._jsc.sc().uiWebUrl().get() + + @property def startTime(self): """Return the epoch time when the Spark Context was started.""" return self._jsc.startTime() --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
