Shawyeok commented on PR #2086: URL: https://github.com/apache/zookeeper/pull/2086#issuecomment-1853223616
> Sorry didn't get chance to get to this until today. Thanks for contributing to this @Shawyeok . I like the idea of fixing the underneath Prometheus quantile computation issue in the prometheus client library. Can you provide some details on the trade off this patch? > > > latency percentiles will be displayed with a relative delay (default is 60 seconds) > > 1. Where does the latency come from? Is it possible to avoid it or reduce it? > > > each Summary metric will have a certain amount of permanent memory overhead > > 2. Do you mean it takes more heap space permanently? How much is the overhead? It would be better to answer both these questions together. Here is the design overview: - Each thread maintains a local buffer (ThreadLocal) to record metrics. - A background thread aggregates all these local buffers and computes the quantile results periodically. Therefore, the latency of monitoring depends on how frequently the background thread aggregates buffers scattered across various threads. This interval is set to 60 seconds by default and is configurable. Currently, it seems to work well in Bookkeeper. The heap space overhead refers to each thread's local buffer. According to the [DoublesSketch][1] documentation, with the default configuration of k (128, approximating an error of about 1.725%), if a latency metric records 65K data points in 60 seconds, each metric will occupy about 2 x 10KB of heap space in its respective thread (the x2 is due to the current/replacement swap strategy). In a real production environment, we observed an instance handling an average of 60K read requests and 600 write requests per second (8 cores with default configuration), where the memory usage for the buffers stayed below 1MB. [1]: https://github.com/apache/datasketches-java/blob/master/src/main/java/org/apache/datasketches/quantiles/DoublesSketch.java -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@zookeeper.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
