Hi Benjamin
Please know there is an ongoing PR for a runner agnostic metrics feature
https://github.com/apache/beam/pull/4548
Le jeudi 02 mars 2017 à 16:26 +0000, Stas Levin a écrit :
> Hi Benjamin,
>
> This is somewhat of a hot topic lately, visibility FTW :)
>
> My experience comes from doing Beam over Spark over Yarn, where we have been
> using Graphite + Grafana (4.0+ gives
> alerting as well).
>
> In terms of metrics, the Spark runner supports both Spark native metrics
> (default + Spark metric sources you
> configure, e.g. JVM), and Beam's, user-defined metrics, all of which are
> reported to whatever Spark sink(s) you have
> configured, in our case Graphite.
>
> There were a few caveats we had to address, most of which had to do with
> associating an application's logical name and
> its application id assigned by Yarn. In other words, making all the metrics
> reported by a given application id, go
> under its logical name required some effort. However, once it's been done it
> allows for dashboards to be built around
> applications' logical names rather than around specific application ids which
> is very useful.
>
> There are more caveats to speak of in the context of metrics and Beam over
> Spark (and streaming in particular), so let
> me know if you find the gory details interesting.
>
> -Stas
>
> On Thu, Mar 2, 2017 at 5:37 PM Amit Sela <[email protected]> wrote:
> > +Stas Levin
> >
> > On Thu, Mar 2, 2017 at 5:30 PM Jean-Baptiste Onofré <[email protected]>
> > wrote:
> > > Hi Benjamin,
> > >
> > > It's a bit related to the Metric discussion on the dev@ mailing list.
> > >
> > > Today, we leverage the monitoring and management provided by the
> > > execution engine of the runner.
> > >
> > > For instance, with the Spark runner, we can use the "regular" history
> > > server, codehale reporter, etc. It's also possible to explicitly use a
> > > sink/fn to send data to a backend (Graphite, elasticsearch, ...).
> > >
> > > We are now discussing about a agnostic way of "pushing" metric to a
> > > backend (something similar to what Karaf Decanter provides).
> > >
> > > Regards
> > > JB
> > >
> > > On 03/02/2017 04:13 PM, Stadin, Benjamin wrote:
> > > > Hi,
> > > >
> > > >
> > > >
> > > > I’m trying to collect a list of open source monitoring and management
> > > > tools for Beam and supported runners.
> > > >
> > > >
> > > >
> > > > What do you use in your daily routine, and what’s your experience?
> > > >
> > > >
> > > >
> > > > Regards
> > > >
> > > > Ben
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > > --
> > > Jean-Baptiste Onofré
> > > [email protected]
> > > http://blog.nanthrax.net
> > > Talend - http://www.talend.com
> > >
