Hi JB, Glad to hear that. Still, I'm thinking about adding support of Meters & Histograms(maybe extending Distribution). As the discussion mentions, problem is that Meter/Histogram cannot be updated directly in current way because their internal data decays after time. Do you plan to refactor current implementation so that they can be supported while working on the generic metric sink?
On Thu, Jun 22, 2017 at 9:37 PM, Jean-Baptiste Onofré <[email protected]> wrote: > Hi > > Agree with Aviem and yes actually I'm working on a generic metric sink. I > created a Jira about that. I'm off today, I will send some details asap. > > Regards > JB > > On Jun 22, 2017, 15:16, at 15:16, Aviem Zur <[email protected]> wrote: > >Hi Cody, > > > >Some of the runners have their own metrics sink, for example Spark > >runner > >uses Spark's metrics sink which you can configure to send the metrics > >to > >backends such as Graphite. > > > >There have been ideas floating around for a Beam metrics sink extension > >which will allow users to send Beam metrics to various metrics > >backends, I > >believe @JB is working on something along these lines. > > > >On Thu, Jun 22, 2017 at 2:00 PM Cody Innowhere <[email protected]> > >wrote: > > > >> Hi guys, > >> Currently metrics are implemented in runners/core as CounterCell, > >> GaugeCell, DistributionCell, etc. If we want to send metrics to > >external > >> systems via metrics reporter, we would have to define another set of > >> metrics, say, codahale metrics, and update codahale metrics > >periodically > >> with beam sdk metrics, which is inconvenient and inefficient. > >> > >> Another problem is that Meter/Histogram cannot be updated directly in > >this > >> way because their internal data decays after time. > >> > >> My opinion would be bridge beam sdk metrics to underlying runners so > >that > >> updates would directly apply to underlying runners (Flink, Spark, > >etc) > >> without conversion. > >> > >> Specifically, currently we already delegate > >> Metrics.counter/gauge/distribution to > >DelegatingCounter/Gauge/Distribution, > >> which uses MetricsContainer to store the actual metrics with the > >> implementation of MetricsContainerImpl. If we can add an API in > >> MetricsEnvironment to allow runners to override the default > >implementation, > >> say, for flink, we have FlinkMetricsContainerImpl, then all metric > >updates > >> will directly apply to metrics in FlinkMetricsContainerImpl without > >> intermediate conversion and updates. And since the metrics are > >> runner-specific, it would be a lot easier to support metrics > >reporters as > >> well as Meters/Histograms. > >> > >> What do you think? > >> >
