Thank you Robert and Alex. I am not running a Beam job in Google Cloud but with Samza Runner, so I am wondering if there is any ETA to add the Histogram metrics in Metrics class so it can be mapped to the SamzaHistogram <http://samza.apache.org/learn/documentation/versioned/api/javadocs/org/apache/samza/metrics/SamzaHistogram.html> metric to the actual emitting.
Best, Ke > On Aug 14, 2020, at 4:44 PM, Alex Amato <[email protected]> wrote: > > One of the plans to use the histogram data is to send it to Google Monitoring > to compute estimates of percentiles. This is done using the bucket counts and > bucket boundaries. > > Here is a describing of roughly how its calculated. > https://stackoverflow.com/questions/59635115/gcp-console-how-are-percentile-charts-calculated > > <https://stackoverflow.com/questions/59635115/gcp-console-how-are-percentile-charts-calculated> > This is a non exact estimate. But plotting the estimated percentiles over > time is often easier to understand and sufficient. > (An alternative is a heatmap chart representing histograms over time. I.e. a > histogram for each window of time). > > > On Fri, Aug 14, 2020 at 4:16 PM Robert Bradshaw <[email protected] > <mailto:[email protected]>> wrote: > You may be interested in the propose histogram metrics: > https://docs.google.com/document/d/1kiNG2BAR-51pRdBCK4-XFmc0WuIkSuBzeb__Zv8owbU/edit > > <https://docs.google.com/document/d/1kiNG2BAR-51pRdBCK4-XFmc0WuIkSuBzeb__Zv8owbU/edit> > > I think it'd be reasonable to add percentiles as its own metric type > as well. The tricky bit (though there are lots of resources on this) > is that one would have to publish more than just the percentiles from > each worker to be able to compute the final percentiles across all > workers. > > On Fri, Aug 14, 2020 at 4:05 PM Ke Wu <[email protected] > <mailto:[email protected]>> wrote: > > > > Hi everyone, > > > > I am looking to add percentile metrics (p50, p90 etc) to my beam job but I > > only find Counter, Gauge and Distribution metrics. I understand that I can > > calculate percentile metrics in my job itself and use Gauge to emit, > > however this is not an easy approach. On the other hand, Distribution > > metrics sounds like the one to go to according to its documentation: "A > > metric that reports information about the distribution of reported > > values.”, however it seems that it is intended for SUM, COUNT, MIN, MAX. > > > > The question(s) are: > > > > 1. is Distribution metric only intended for sum, count, min, max? > > 2. If Yes, can the documentation be updated to be more specific? > > 3. Can we add percentiles metric support, such as Histogram, with > > configurable list of percentiles to emit? > > > > Best, > > Ke
