sijie opened a new issue #1078: Prometheus stats provider doesn't cache stats logger objects URL: https://github.com/apache/bookkeeper/issues/1078 *Problem* while analyzing a directbuffer oom problem in pulsar cluster, we noticed that the generation of prometheus objects is too frequent, the amount of prometheus objects is several times of other objects. ``` num #instances #bytes class name ---------------------------------------------- 1: 120041360 3841323520 io.prometheus.client.CKMSQuantiles$Item 2: 120042101 2881010424 java.util.LinkedList$Node 3: 29234508 935504256 io.netty.util.Recycler$DefaultHandle 4: 5326 481710096 [J 5: 5442536 478943168 io.netty.buffer.PooledUnsafeDirectByteBuf 6: 3948006 221088336 org.apache.bookkeeper.bookie.Journal$QueueEntry ``` ``` num #instances #bytes class name ---------------------------------------------- 1: 34108130 1091460160 io.prometheus.client.CKMSQuantiles$Item 2: 34108849 818612376 java.util.LinkedList$Node 3: 6537 481896128 [J 4: 8300630 265620160 io.netty.util.Recycler$DefaultHandle 5: 75884 150881688 [I ``` It turns out that unlike other stats provider implementation, the prometheus stats provider doesn't have any caching mechanism built in. for each `scope`, `getOpStatsLogger` and `getCounter` call, it will return a brand new object. This would cause it generate several stats object for each add/read entry calls. This brings in a burden to the bookie jvm, and would cause other issue on bookie, for example, might cause the direct buffer building up and cause jvm oom. *Solution* We can use the `CachingStatsProvider` to wrap the prometheus stats provider. so it would have the caching capability. `TwitterOstrichProvider` is a good example to follow : https://github.com/apache/bookkeeper/blob/master/bookkeeper-stats-providers/twitter-ostrich-provider/src/main/java/org/apache/bookkeeper/stats/twitter/ostrich/OstrichProvider.java#L72
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
