-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 All,
I'm trying to get into the business of inspecting my JVM using the mbeans that Tomcat (and the JVM) expose, but I'm having great difficulty. I'm not even sure if I'm doing the right things. I have a running Tomcat instance (happens to be 7.0.25 running on Sun JRE 1.6.something on Mac OS X Lion), and I can happily connect to it using jconsole (using a process id) with no special startup parameters for Tomcat itself. If I understand this correctly, it's because when the JVM starts up, it allows localhost JMX inspection with no authentication using the "Attach" API [1]. I have enabled neither Tomcat's mbeans.GlobalResourcesLifecycleListener not the JmxRemoveLifecycleListener [2]. I can see a wealth of information about the JVM and Tomcat in this configuration. Now I'm trying to get similar information using a command-line tool that is very simple called check_jmx -- it's a plug-in for Nagios. It appears that this tool does not support the "attach" API and so it looks like I'll have to enable "remote JMX", so I've followed the instructions on Tomcat's monitoring page to enable remote JMX [3]: $ export CATALINA_OPTS='\ - -Dcom.sun.management.jmxremote \ - -Dcom.sun.management.jmxremote.port=1234 \ - -Dcom.sun.management.jmxremote.ssl=false \ - -Dcom.sun.management.jmxremote.authenticate=false' When running Tomcat, I can see that those parameters are actually part of the command line. I can also see that the port is bound: $ netstat -an | grep 1234 tcp46 0 0 *.1234 *.* LISTEN tcp4 0 0 127.0.0.1.12345 *.* LISTEN tcp6 0 0 ::1.12345 *.* LISTEN I can use this tool with a command line such as the following: $ ./check_jmx -U service:jmx:rmi:///jndi/rmi://localhost:1234/jmxrmi \ -O java.lang:type=Memory -A HeapMemoryUsage -K used \ -I HeapMemoryUsage -J used -vvvv -w 4248302272 -c 5498760192 JMX OK HeapMemoryUsage.used=35587776{committed=85000192;init=0;max=129957888;used=35587776} Without going into too much detail, those options allow check_jmx to probe a certain value, then use the -w and -c options to determine whether the value obtained from the JMX server is okay, in the warning zone, or in the critical zone, and then builds a text response based upon that. So, now I want to move to my server which happens to be an Amazon EC2 instance with an elastic IP assigned to it so I can get to it from the outside. Most ports are blocked. I try the same type of thing: set up the com.sun.management.etc parameters and launch Tomcat. netstat reports: $ netstat -plan | grep 1234 tcp6 0 0 :::1234 :::* LISTEN 2819/java So I should be good to go. I launch the command the same as on my local machine, but this time it hangs and then I get an exception: $ ./check_jmx -U service:jmx:rmi:///jndi/rmi://localhost:1234/jmxrmi - -O java.lang:type=Memory -A HeapMemoryUsage -K used -I HeapMemoryUsage - -J used -vvvv -w 4248302272 -c 5498760192 JMX CRITICAL Connection refused to host: [public IP]; nested exception is: java.net.ConnectException: Connection timed out connecting to java.lang:type=Memory by URL service:jmx:rmi:///jndi/rmi://localhost:1234/jmxrmijava.rmi.ConnectException: Connection refused to host: 107.21.113.203; nested exception is: java.net.ConnectException: Connection timed out at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601) at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198) at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:110) at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source) at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2329) at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:279) at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248) at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:207) at org.nagios.JMXQuery.connect(JMXQuery.java:53) at org.nagios.JMXQuery.main(JMXQuery.java:75) Caused by: java.net.ConnectException: Connection timed out at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at java.net.Socket.<init>(Socket.java:375) at java.net.Socket.<init>(Socket.java:189) at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22) at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128) at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595) ... 10 more So, [public IP] is actually my public, IPv4 address. I figure the problem might be IPv4-versus-IPv6, but before I go "preferring" the IPv4 stack, I check netstat during the execution of check_jmx: $ netstat -plan | grep 1234 tcp6 0 0 :::1234 :::* LISTEN 2819/java tcp6 0 0 127.0.0.1:40292 127.0.0.1:1234 ESTABLISHED 2950/java tcp6 0 0 127.0.0.1:1234 127.0.0.1:40292 ESTABLISHED 2819/java (pid 2819 is my Tomcat process, and 2950 is check_jmx). So, what else is check_jmx doing? $ netstat -plan | grep 2950 (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) tcp6 0 0 127.0.0.1:40292 127.0.0.1:1234 ESTABLISHED 2950/java tcp6 0 1 10.192.214.127:44419 107.21.113.203:51087 SYN_SENT 2950/java So, check_jmx is actually successfully establishing a socket with the JMX port but it looks like it's doing something weird with the (random) RMI port. First, it's trying to use the public IP address instead of just localhost (which seems stupid, but that's probably what the JMX service told it to do). Second, it's calling-out on the internet IP address, which is probably a result of using a non-localhost destination IP address. So, a couple of questions: 1. Is there a way to tell the JVM which interface to use for the second, random RMI port that gets opened? I think if everything runs on :: or 127.0.0.1 then all will probably be well. 2. Should I just give up and use JmxRemoteLifecycleListener? Will that actually buy me anything? I will always be running check_jmx locally. I'm not s huge fan of having to deploy with an optional library tossed-into Tomcat's lib direcory (even though it's actually from the find Apache folks). 3. Should I just give up and use the manager app's jmxproxy? I don't currently deploy the manager app, and I'd like to avoid doing that if possible. But, it may be a slightly cleaner solution. 4. Should I hack the code for check_jmx to use the Attach API and try to avoid all of this stupid port business? Getting the PID of the Tomcat process shouldn't be hard as long as I use CATALINA_PID and get the value from there. Those playing along at home might remember that I recently posted a question about Kitty, another JMX CLI program. The documentation for Kitty has the same type of ugly RMI URLs in it, so I suspect I'll have the same problem with another tool: I think this is squarely a server-process configuration issue that I have to deal with. Any suggestions would be most appreciated. Thanks, - -chris [1] http://docs.oracle.com/javase/6/docs/technotes/guides/management/agent.html [2] http://tomcat.apache.org/tomcat-7.0-doc/config/listeners.html#JMX_Remote_Lifecycle_Listener_-_org.apache.catalina.mbeans.JmxRemoteLifecycleListener [3] http://tomcat.apache.org/tomcat-7.0-doc/monitoring.html#Enabling_JMX_Remote -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk8hnDcACgkQ9CaO5/Lv0PCFqgCgm0Ai/TITuEPzCIuwZvRF7WGR SzsAmgKtgVD4Mzv4QBs4Yq4UycXwuRDd =V/nO -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org