You can check out Spree <https://github.com/hammerlab/spree> for one data point about how this can be done; it is a near-clone of the Spark web UI that updates in real-time.
It uses JsonRelay <https://github.com/hammerlab/spark-json-relay>, a SparkListener that sends events as JSON over the network; it receives those events, aggregates stats similar to the JobProgressListener and writes those to Mongo in slim <https://github.com/hammerlab/slim>, and then Spree uses Meteor to display a real-time web UI based on the data in Mongo. On Mon, Sep 14, 2015 at 2:18 AM Prashant Sharma <scrapco...@gmail.com> wrote: > Hi all, > > TLDR; > Some of my colleagues at Imaginea are interested in building an alternate > UI for Spark. Basically allow people or groups to build an alternate UI for > Spark. > > More Details: > Looking at feasibility, it feels definitely possible to do. But we need a > consensus on a public(can be experimental initially ) interface which would > give access to UI in core. Given this is done, their job will be easy. > > Infact, it opens up a lot of possibilities for alternate UI for Apache > spark. Also considering a pluggable UI - where alternate UI can just be a > plugin. Ofcourse, implementing later can be a long term goal. Elasticsearch > is a good example of the later approach. > > My knowledge on this is certainly limited. Comments and criticism > appreciated. > > Thanks, > Prashant >
