On Fri, 5 Jun 2020 at 12:31, Cameron Kerr <[email protected]> wrote:
> > > On Friday, 5 June 2020 22:31:13 UTC+12, Brian Brazil wrote: >> >> On Fri, 5 Jun 2020 at 11:13, Cameron Kerr <[email protected]> wrote: >> >>> Hi all, I've spent a few days coming up to speed on understanding >>> jmx_exporter, and I think I'm in a pretty good place now, understanding how >>> MBeans, JMX, and RMI work. >>> >>> I've so far deployed jmx_exporter in two ways: >>> >>> * as a Java Agent on Tomcat 6 (on RHEL6) for a small application that >>> includes SOLR and PostgreSQL >>> * and as a standalone Java HTTP server on Tomcat 8.5.34 (that comes >>> bundled with a mission-critical application) >>> >>> I found going with the Java Agent relatively easy, although I think I'll >>> contribute a blog post and pull-request to help on the documentation front. >>> >>> You might reasonably ask why I'm bothering with the HTTP server. Here's >>> my business logic that drives this: >>> >>> * we have a mission-critical application that we urgently need to >>> improve our visibility on to diagnose some performance limits we believe >>> we're reaching >>> * we're reluctant to introduce (and cause an outage to introduce) a java >>> agent --- as far as I'm aware jmx_exporter lacks the ability that jconsole >>> has to dynamically inject an agent. >>> >> >> It's a very simple Java agent that doesn't do anything fancy. I believe >> it's possible if you already know how to do such things, though using it as >> an agent is almost always better. >> > > I can appreciate that a lot more now that I understand more about the > issues around RMI etc. > > >> * as part of a previous monitoring drive, we've already introduced the >>> appropriate Remote JMX configuration (-D....jmxremote... etc.), which means >>> we can introduce some monitoring into our production environment and easily >>> restart the JMX exporter as needed to iterate through configuration changes. >>> >> >> The JMX exporter will pick up configuration changes without restarting >> > > Very happy to hear that. I'll be sure to mention that in an upcoming doc > PR, as that behaviour is not listed on the README.md that I could find. > > >> [...] You also lose some process and JVM metrics. >> > > I'm not sure that's necessarily the case; at least I can see MBeans for > Memory, OperatingSytem, Runtime, Threading, etc. under the 'java.lang' > domain, but that's likely to be JVM dependent, I don't think I saw that on > Java 7 (or Tomcat 6). But I guess you mean that jmx_exporter is exporting > some of its own information, rather than from MBeans for "process and JVM > metrics" > There's some process stuff it'll pull in on Linux systems that's not available from mBeans. It shouldn't vary by Java version. > > I would like to propose that we introduce one of two things: >>> >>> EITHER add a new attribute whitelistObjectNameAttributes that could be >>> used for Jconsole-style attribute at a time (or similar; can you grab a few >>> named attributes in one go?), which would allow for either the broad-brush >>> or fine-brush approach to collecting the data; >>> >> >> I'd be open to considering an attribute name blacklist, if there's a sane >> way to specify that. >> > > Looking at the following makes me think it should be reasonably > straightforward to create an additional white/blacklists for Attributes > within an ObjectName; I think the crux of it would be in located around > this part of the code: > > > https://github.com/prometheus/jmx_exporter/blob/master/collector/src/main/java/io/prometheus/jmx/JmxScraper.java#L140-L147 > Map<String, MBeanAttributeInfo> name2AttrInfo = new > LinkedHashMap<String, MBeanAttributeInfo>(); > for (int idx = 0; idx < attrInfos.length; ++idx) { > MBeanAttributeInfo attr = attrInfos[idx]; > if (!attr.isReadable()) { > logScrape(mbeanName, attr, "not readable"); > continue; > } > name2AttrInfo.put(attr.getName(), attr); > > Plus there would have to the config-level stuff and validation thereof, > and plumbing the config into the various method calls in places where where > whitelistObjectName and blacklistObjectName appear. And test-cases. > That's the bit of code, the challenge is that there's more than one object so you can't work off attribute name alone. Brian > > On a higher level, I'd suggest looking at ways to get metrics that don't >> involve JMX. Using client_java directly as far as possible will avoid all >> the fun and performance issues that JMX brings. >> > > That would seem to be impractical in all my use-cases, as I don't control > the application (certainly not the code-base) to that extent, and JMX would > be the right, supported, tool for this job. I think I can probably work in > the agent though; it seems to be the normal way for other similar vendors > to do the same monitoring thing. > > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/cda0dce5-603c-4af8-be79-712f7f2a6436o%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/cda0dce5-603c-4af8-be79-712f7f2a6436o%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- Brian Brazil www.robustperception.io -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAHJKeLpfO0jME6Hx8fgudS4eFw49hufUd7jyVOGX_XhJpZG7zA%40mail.gmail.com.
