Hi Peter, Thank you for your help, I got it to work!
I have added the lines below to the tomcat service file now and it works. A separate JMX process is now started for each of the apache tomcat instances I start, this could cause some port conflicts though - looks like a new challange. #Environment="JAVA_OPTS=-Djava.security.egd=file:///dev/urandom -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=10080 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false" Environment="JAVA_OPTS=-Djava.security.egd=file:///dev/urandom" Environment="CATALINA_PID=/var/tomcat/%i/run/tomcat.pid" Environment="CATALINA_BASE=/var/tomcat/%i/" Environment="CATALINA_HOME=/opt/tomcat/10/" Environment="CATALINA_OPTS=-Xms512M -Xmx1024M -server -XX:+UseParallelGC -javaagent:/var/tomcat/tomcat10/lib/jmx_prometheus_javaagent-0.17.2.jar=10082:/var/tomcat/tomcat10/prometheus/config.yml" Another challenge is to get it to configure it to HTTPS and HTTP, but I think ill be able to manage that by adding --web.config.file to the configuration. Kind regards, Melvin ________________________________ From: l...@kreuser.name <l...@kreuser.name> Sent: Monday, January 16, 2023 6:05 PM To: Tomcat Users List <users@tomcat.apache.org> Subject: Re: Prometheus on Apache Tomcat multiple tomcat Instances Melvin, > Am 16.01.2023 um 14:52 schrieb Melvin Oosterveen > <melvinoosterv...@hotmail.com>: > > > Hi all, > > I'm currently working on Apache Tomcat stacks within my company and I would > like to integrate Prometheus into our deployments. We have decided to go for > the Advanced Configuration - Multiple Tomcat Instances on UNIX systems > described in the RUNNING.txt. > > Does anyone have some experience with the setup of the JMX Prometheus > exporter using multiple Tomcat instances? My goal is to have a separate Java > process (Prometheus) started when I start an Apache Tomcat instance. > in my case the prometheus java agent is tied 1:1 to a tomcat process. In my understanding, that would mean 1 prometheus exporter per tomcat (each on a different port of course). > So far I have downloaded the JMX > exporter<https://github.com/prometheus/jmx_exporter> and added it to the > created Tomcat Instance with a minimal Prometheus configuration. > As per the github page jmxagent is the recommended way and not the standalone http server (which queries tomcat remotely on the jmx port). Can you explain which one you are really trying to use? > ```bash > > [root@melvin prometheus]# cat config.yml > > hostPort: localhost:9999 is tomcat listening on this port with jmx? You are trying to query jmx on this host on this port (your tomcat process). But did you start the prometheus exporter as a separate java process? java -jar jmx_prometheus_httpserver-0.17.2.jar 12345 config.yaml prometheus >> exporter:12345 >> tomcat:9999 (jmx) > > rules: > > - pattern: ".*" > > ``` > > I have created a service file which allows me to start the Apache Tomcat > instances separately from each other. Inside this configuration I tried to > start the Prometheus Java process, but so far I have failed and the Java > process won't expose itself on the configured port. > ```bash > > [root@melvin prometheus]# cat /etc/systemd/system/tomcat-10@.service > > [Unit] > > Description=Tomcat - instance %i > > After=syslog.target network.target > > > [Service] > > Type=forking > > > User=tomcat > > Group=tomcat > > > WorkingDirectory=/var/tomcat/%i > > > Environment="JAVA_HOME=/etc/alternatives/jre_openjdk" > > Environment="JAVA_OPTS=-Djava.security.egd=file:///dev/urandom > -Djavaagent:/var/tomcat/tomcat10/lib/jmx_prometheus_javaagent-0.17.2.jar=10080:/var/tomcat/tomcat10/prometheus/config.yml" At this point you are using the jmxagent - inside the tomcat process. Then you MUST leave out the hostPort. Did you try to curl localhost:10080 ? prometheus >> tomcat:10080 >> internal jmx lookup If you really want to use the external exporter, you will have to configure jmx in JAVA_OPTS instead to run on port 9999. I'm not sure if the following is enough or too much: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.rmi.port=10002 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false > Environment="CATALINA_PID=/var/tomcat/%i/run/tomcat.pid" > > Environment="CATALINA_BASE=/var/tomcat/%i/" > > Environment="CATALINA_HOME=/opt/tomcat/10/" > > Environment="CATALINA_OPTS=-Xms512M -Xmx1024M -server -XX:+UseParallelGC" > > > ExecStart=/opt/tomcat/10/bin/startup.sh > > ExecStop=/opt/tomcat/10/bin/shutdown.sh > > > #RestartSec=10 > > #Restart=always > > > [Install] > > WantedBy=multi-user.target > > ``` > > The instance then starts without errors and starts my sample.war application > which I have downloaded from Apache.org. The initiated Java process is even > shown. > ```bash > > tomcat 115910 1 0 Jan12 ? 00:02:24 > /etc/alternatives/jre_openjdk/bin/java > -Djava.util.logging.config.file=/var/tomcat/tomcat10//conf/logging.properties > -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager > -Djava.security.egd=file:///dev/urandom > -Djavaagent:/var/tomcat/tomcat10/lib/jmx_prometheus_javaagent-0.17.2.jar=10080:/var/tomcat/tomcat10/prometheus/config.yml > -Djdk.tls.ephemeralDHKeySize=2048 > -Djava.protocol.handler.pkgs=org.apache.catalina.webresources > -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Xms512M -Xmx1024M > -server -XX:+UseParallelGC -Dignore.endorsed.dirs= -classpath > /opt/tomcat/10//bin/bootstrap.jar:/var/tomcat/tomcat10//bin/tomcat-juli.jar > -Dcatalina.base=/var/tomcat/tomcat10/ -Dcatalina.home=/opt/tomcat/10/ > -Djava.io.tmpdir=/var/tomcat/tomcat10//temp > org.apache.catalina.startup.Bootstrap start > > ``` > > I wonder if anyone could help me out with the right setup of Prometheus on > Apache Tomcat multiple tomcat Instances. > I would go the javaagent route and get one tomcat fixed - then all others will work easily. To get authentication right, put a haproxy (or an Apache httpd if it already exists) in front of your exporter, so you can limit queries to your prometheus host. > Thanks in advance, > > Melvin Oosterveen Hope this helps. Peter
