Hi, Our intention was to minimize the scope of the desired effect to only those strings which are used in the caching mechanism of the JMX exporter. UseStringDeduplication is a global setting that affects the main application as well and this is undesirable in our case.
If there are other solutions to the problem at hand, I'll be glad to explore them, of course. P.S. Doug, when do you think this change will be available in a release, be it minor or major? Kind regards, Yana On Friday, December 20, 2024 at 1:57:27 AM UTC+2 Francisco Melo junior wrote: > Sure thank you. > > Can you be more specific on the consequences? > > Evidently there will be an overhead for the comparison for each string, if > that’s what you mean. But if the focus is less footprint. And should be > (almost) similar to intern(), because the options are done the same number > of times. > > Also it should only remove the duplicates but keep the strings all in the > pool. > > Unless the strings are used for as the actual keys in the caching so > removing the duplicates will indeed be problematic because you will lose > access to the value associated with it. > > Sent from Earth > > On Dec 19, 2024, at 3:19 PM, Doug Hoard <doug....@gmail.com> wrote: > > Francisco, > > > Using "XX:+UseStringDeduplication" to enable String deduplication for the > whole JVM process could have unwanted side effects, so performing the > "intern()" of the cache keys directly was deemed a much safer option. > > On Wednesday, December 18, 2024 at 7:30:48 AM UTC-5 Francisco Melo junior > wrote: > >> Hello I was wondering here what is the result of you disable string >> deduplication XX:+UseStringDeduplication, or was that the caching you >> did? It should reduce the footprint considerably if the duplication is the >> problem, not increase it. >> >> -Francisco >> >> On Dec 13, 2024, at 9:54 AM, Яна Илиева <yana.il...@gmail.com> wrote: >> >> Hello Doug, >> >> >> Thank you for the quick reaction and the testing. I will proceed with >> opening the PR. >> >> Kind regards, >> Yana >> On Friday, December 13, 2024 at 3:44:59 PM UTC+2 Doug Hoard wrote: >> >>> Yana, >>> >>> The full integration test passed without issues. Can you create a PR of >>> your findings and the change so we can document it and give credit? >>> >>> -Doug >>> >>> On Friday, December 13, 2024 at 1:44:47 AM UTC-5 Doug Hoard wrote: >>> >>>> Using String intern() can help in some scenarios and cause performance >>>> issues in other scenarios. >>>> >>>> I made your change on my development branch and ran a quick test >>>> (./quick-test.sh) and didn't see any unit test or integration test issues. >>>> >>>> I just started a full regression test and will have some results in the >>>> morning (US time.) (It takes a little over 3 hours.) >>>> >>>> -Doug >>>> >>>> On Thursday, December 12, 2024 at 11:02:02 AM UTC-5 Яна Илиева wrote: >>>> >>>>> Hello, >>>>> >>>>> While using the Prometheus JMX Exporter as a Java agent in our >>>>> application, we observed frequent GC clean ups and it turns out it is >>>>> because the available heap space got exhausted by the caching of the >>>>> Prometheus JMX Exporter. >>>>> >>>>> For context - we have an exporter yaml file with about 40 rules >>>>> defined and there are many JMX MBeans which we process in order to get >>>>> the >>>>> required metrics. Without caching enabled for the rules, the request can >>>>> take a minimum of 1 minute to complete. In an attempt to reduce this >>>>> time, >>>>> we enabled caching and observed that the MatchedRulesCache object can >>>>> take >>>>> around 600 MB in a particular case. There is a realistic potential in out >>>>> case that this object grows above 1 GB, which is a huge amount of space >>>>> for >>>>> this kind of process. >>>>> >>>>> We identified that the major reason for the large size of >>>>> MatchedRulesCache is that it contains duplicating String objects for the >>>>> same MBean names. The object keeps all MBean names the rules were matched >>>>> against previously, in order to avoid expensive pattern matching later. >>>>> Although each rule caches the same set of MBean names, those are in fact >>>>> separate objects in the heap. >>>>> >>>>> The origin of the matter was found in >>>>> https://github.com/prometheus/jmx_exporter/blob/a3dac9acee1464531cd87502579178a1fec1cc76/collector/src/main/java/io/prometheus/jmx/JmxCollector.java#L584 >>>>> >>>>> where for each rule with caching enabled, there is a new String object >>>>> created, although such could already exist in memory from a previous >>>>> iteration. >>>>> >>>>> The duplication is by a factor of the number of rules with caching >>>>> enabled. >>>>> >>>>> I experimented with interning the String, >>>>> >>>>> *String matchName = (beanName + attributeName + ": " + >>>>> matchBeanValue).intern();* >>>>> >>>>> so that the JVM's string pool is utilized and the String objects >>>>> reused. The result in speed and heap space used by cache is described >>>>> below: >>>>> >>>>> [image: Screenshot from 2024-12-12 14-53-26.png] >>>>> >>>>> What do you think about this and do you find this suggestion could >>>>> have negative consequences in certain cases? >>>>> >>>>> Kind regards, >>>>> Yana Ilieva >>>>> >>>> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to prometheus-devel...@googlegroups.com. >> To view this discussion visit >> https://groups.google.com/d/msgid/prometheus-developers/6c2d05ad-6674-424e-afe3-3289b370b286n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-developers/6c2d05ad-6674-424e-afe3-3289b370b286n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> -- > You received this message because you are subscribed to the Google Groups > "Prometheus Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to prometheus-devel...@googlegroups.com. > > To view this discussion visit > https://groups.google.com/d/msgid/prometheus-developers/b8828a92-6422-4fb6-9103-e038e1c4b191n%40googlegroups.com > > <https://groups.google.com/d/msgid/prometheus-developers/b8828a92-6422-4fb6-9103-e038e1c4b191n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/prometheus-developers/c5f49dc1-b279-434a-af65-374e9b3d61d2n%40googlegroups.com.
