Is SparkSession.getActiveSession what you're looking for? On Tue, Aug 7, 2018 at 5:11 PM Andrew Melo <andrew.m...@gmail.com> wrote:
> Hello, > > One pain point with various Jupyter extensions [1][2] that provide > visual feedback about running spark processes is the lack of a public > API to introspect the web URL. The notebook server needs to know the > URL to find information about the current SparkContext. > > Simply looking for "localhost:4040" works most of the time, but fails > if multiple spark notebooks are being run on the same host -- spark > increments the port for each new context, leading to confusion when > the notebooks are trying to probe the web interface for information. > > I'd like to implement an analog to SparkContext.getOrCreate(), perhaps > called "getIfExists()" that returns the current singleton if it > exists, or None otherwise. The Jupyter code would then be able to use > this entrypoint to query Spark for an active Spark context, which it > could use to probe the web URL. > > It's a minor change, but this would be my first contribution to Spark, > and I want to make sure my plan was kosher before I implemented it. > > Thanks! > Andrew > > > > > > [1] https://krishnan-r.github.io/sparkmonitor/ > > [2] https://github.com/mozilla/jupyter-spark > > --------------------------------------------------------------------- > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >
