Thanks Sivakumaran I don't think we can use Zeppelin for this purpose. It is not a real time dashboard or can it be. I use it but much like Tableau with added Scala programming.
Does anyone know of open source real time dashboards? Cheers Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On 27 August 2016 at 09:42, Sivakumaran S <siva.kuma...@icloud.com> wrote: > I would love to participate in developing a dashboard of some sort in lieu > (or at least complement it) of Spark UI . > > Regards, > > Sivakumaran S > > On 27 Aug 2016 9:34 a.m., Mich Talebzadeh <mich.talebza...@gmail.com> > wrote: > > Are we actually looking for a eal time dashboard of some sort for Spark UI > interface? > > After all one can think a real time dashboard can do this! > > HTH > > > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpress.com > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > On 26 August 2016 at 23:38, Mich Talebzadeh <mich.talebza...@gmail.com> > wrote: > > Thanks Jacek, > > I will have a look. I think it is long overdue. > > I mean we try to micro batch and stream everything below seconds but when > it comes to help monitor basics we are still miles behind :( > > Cheers, > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpress.com > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > On 26 August 2016 at 23:21, Jacek Laskowski <ja...@japila.pl> wrote: > > Hi Mich, > > I don't think so. There is support for a UI page refresh but I haven't > seen it in use. > > See StreamingPage [1] where it schedules refresh every 5 secs, i.e. > Some(5000). In SparkUIUtils.headerSparkPage [2] there is > refreshInterval but it's not used in any place in Spark. > > Time to fill an JIRA issue? > > What about REST API and httpie updating regularly [3]? Perhaps Metrics > with ConsoleSink [4]? > > [1] https://github.com/apache/spark/blob/master/streaming/src/ma > in/scala/org/apache/spark/streaming/ui/StreamingPage.scala#L158 > [2] https://github.com/apache/spark/blob/master/core/src/main/sc > ala/org/apache/spark/ui/UIUtils.scala#L202 > [3] http://spark.apache.org/docs/latest/monitoring.html#rest-api > [4] http://spark.apache.org/docs/latest/monitoring.html#metrics > > Pozdrawiam, > Jacek Laskowski > ---- > https://medium.com/@jaceklaskowski/ > Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark > Follow me at https://twitter.com/jaceklaskowski > > > On Thu, Aug 25, 2016 at 11:55 AM, Mich Talebzadeh > <mich.talebza...@gmail.com> wrote: > > Hi, > > > > This may be already there. > > > > A spark job opens up a UI on port specified by --conf > "spark.ui.port=${SP}" > > that defaults to 4040. > > > > However, on UI one needs to refresh the page to see the progress. > > > > Can this be polled so it is refreshed automatically > > > > Thanks > > > > > > Dr Mich Talebzadeh > > > > > > > > LinkedIn > > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJ > d6zP6AcPCCdOABUrV8Pw > > > > > > > > http://talebzadehmich.wordpress.com > > > > > > Disclaimer: Use it at your own risk. Any and all responsibility for any > > loss, damage or destruction of data or any other property which may arise > > from relying on this email's technical content is explicitly disclaimed. > The > > author will in no case be liable for any monetary damages arising from > such > > loss, damage or destruction. > > > > > > > > >
