gortiz opened a new pull request, #11695: URL: https://github.com/apache/pinot/pull/11695
As we know, Pinot uses Yammer as default metric exported, which has several issues. The main issue, explained in https://github.com/apache/pinot/pull/9416, is that Yammer histograms (including Timers) can only use the default exponential decaying reservoir, which returns incorrect results when sample data doesn't have a normal distribution. See the following post to understand the issue: - https://medium.com/expedia-group-tech/your-latency-metrics-could-be-misleading-you-how-hdrhistogram-can-help-9d545b598374 (very recommended reading) - https://gist.github.com/1010sachin/8903fdcdfd56454bd57ec6ddfa56ec47) As explained in the these two blogposts, dropwizard has provides the ability to change the reservoir and there are two well known alternatives: - SlidingTimeWindowArrayReservoir, included in Dropwizard > 3.2.3 - HdrHistogramReservoir, a third party reservoir that uses the well known [hdr histogram](http://hdrhistogram.org/) library under the hood. I have experience using the later and it works very well. But what I've done in this PR is to modify `pinot-dropwizard` to use SlidingTimeWindowArrayReservoir by default. I've discarded the other two options for the following reasons: - Default exponential decaying reservoir: It is very bad in a lot of cases. See the first mentioned post. This affects us in query latency metrics when tables are not frequently queried. - HdrHistogramReservoir: Although my personal experience with the library is very good, I decided to do not use it in order to do not add just another dependency to Pinot. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
