NodeTool won't connect remotely
Hi, I'm trying to manage my cassandra cluster from a remote box and having issues getting nodetool to connect. All the machines I'm using are running on AWS. Here's what happens when I try: /opt/apache-cassandra-0.6.4/bin/nodetool -h xxx.xxx.xxx.143 -p 10036 ring Error connecting to remote JMX agent! java.rmi.ConnectException: Connection refused to host: xxx.xxx.xxx.143; nested exception is: java.net.ConnectException: Connection timed out When I'm local to a box (Ubuntu 10.04) running Cassandra, I can connect fine via both 127.0.0.1 and external ip (xxx.xxx.xxx.143). I can telnet into the jmx port from an external machine fine: telnet xxx.xxx.xxx.143 10036 Trying xxx.xxx.xxx.143... Connected to xxx.xxx.xxx.143. Escape character is '^]'. I already added the -Djava.rmi.server.hostname parameter to the java runtime, but it didn't seem to affect anything. /usr/bin/jsvc -home /usr/lib/jvm/java-6-openjdk/jre -pidfile /var/run/cassandra.pid -errfile 1 -outfile /var/log/cassandra/output.log -cp /usr/share/cassandra/antlr-3.1.3.jar:/usr/share/cassandra/apache-cassandra-0.6.3.jar:/usr/share/cassandra/avro-1.2.0-dev.jar:/usr/share/cassandra/clhm-production.jar:/usr/share/cassandra/commons-cli-1.1.jar:/usr/share/cassandra/commons-codec-1.2.jar:/usr/share/cassandra/commons-collections-3.2.1.jar:/usr/share/cassandra/commons-lang-2.4.jar:/usr/share/cassandra/google-collections-1.0.jar:/usr/share/cassandra/hadoop-core-0.20.1.jar:/usr/share/cassandra/high-scale-lib.jar:/usr/share/cassandra/ivy-2.1.0.jar:/usr/share/cassandra/jackson-core-asl-1.4.0.jar:/usr/share/cassandra/jackson-mapper-asl-1.4.0.jar:/usr/share/cassandra/jline-0.9.94.jar:/usr/share/cassandra/json-simple-1.1.jar:/usr/share/cassandra/libthrift-r917130.jar:/usr/share/cassandra/log4j-1.2.14.jar:/usr/share/cassandra/slf4j-api-1.5.8.jar:/usr/share/cassandra/slf4j-log4j12-1.5.8.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar -Xmx4G -Xms128M -Djava.rmi.server.hostname=xxx.xxx.xxx.143 -Dcassandra -Dstorage-config=/etc/cassandra -Dcom.sun.management.jmxremote.port=10036 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false org.apache.cassandra.thrift.CassandraDaemon netstat shows that I'm still bound to IP6 netstat -nap|grep 10036 tcp6 0 0 :::10036:::*LISTEN 29277/jsvc And, now I'm at an impasse. Any help would be greatly appreciated. Thanks -Allan
Re: NodeTool won't connect remotely
I think that JMX needs additional ports to function correctly. Try to disable all firewalls between the client and the server so that client can connect to any port in the server and try again. - Juho Mäkinen On Mon, Aug 30, 2010 at 7:07 PM, Allan Carroll alla...@gmail.com wrote: Hi, I'm trying to manage my cassandra cluster from a remote box and having issues getting nodetool to connect. All the machines I'm using are running on AWS. Here's what happens when I try: /opt/apache-cassandra-0.6.4/bin/nodetool -h xxx.xxx.xxx.143 -p 10036 ring Error connecting to remote JMX agent! java.rmi.ConnectException: Connection refused to host: xxx.xxx.xxx.143; nested exception is: java.net.ConnectException: Connection timed out When I'm local to a box (Ubuntu 10.04) running Cassandra, I can connect fine via both 127.0.0.1 and external ip (xxx.xxx.xxx.143). I can telnet into the jmx port from an external machine fine: telnet xxx.xxx.xxx.143 10036 Trying xxx.xxx.xxx.143... Connected to xxx.xxx.xxx.143. Escape character is '^]'. I already added the -Djava.rmi.server.hostname parameter to the java runtime, but it didn't seem to affect anything. /usr/bin/jsvc -home /usr/lib/jvm/java-6-openjdk/jre -pidfile /var/run/cassandra.pid -errfile 1 -outfile /var/log/cassandra/output.log -cp /usr/share/cassandra/antlr-3.1.3.jar:/usr/share/cassandra/apache-cassandra-0.6.3.jar:/usr/share/cassandra/avro-1.2.0-dev.jar:/usr/share/cassandra/clhm-production.jar:/usr/share/cassandra/commons-cli-1.1.jar:/usr/share/cassandra/commons-codec-1.2.jar:/usr/share/cassandra/commons-collections-3.2.1.jar:/usr/share/cassandra/commons-lang-2.4.jar:/usr/share/cassandra/google-collections-1.0.jar:/usr/share/cassandra/hadoop-core-0.20.1.jar:/usr/share/cassandra/high-scale-lib.jar:/usr/share/cassandra/ivy-2.1.0.jar:/usr/share/cassandra/jackson-core-asl-1.4.0.jar:/usr/share/cassandra/jackson-mapper-asl-1.4.0.jar:/usr/share/cassandra/jline-0.9.94.jar:/usr/share/cassandra/json-simple-1.1.jar:/usr/share/cassandra/libthrift-r917130.jar:/usr/share/cassandra/log4j-1.2.14.jar:/usr/share/cassandra/slf4j-api-1.5.8.jar:/usr/share/cassandra/slf4j-log4j12-1.5.8.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar -Xmx4G -Xms128M -Djava.rmi.server.hostname=xxx.xxx.xxx.143 -Dcassandra -Dstorage-config=/etc/cassandra -Dcom.sun.management.jmxremote.port=10036 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false org.apache.cassandra.thrift.CassandraDaemon netstat shows that I'm still bound to IP6 netstat -nap|grep 10036 tcp6 0 0 :::10036 :::* LISTEN 29277/jsvc And, now I'm at an impasse. Any help would be greatly appreciated. Thanks -Allan
Re: NodeTool won't connect remotely
Thanks! That did it. Looks like the connection happens on 10036 and then the server negotiates a separate port for continued communication. Found this article once I knew what to look for. It also describes how to get more consistency on port numbers to allow for ssh tunneling and firewalls. From http://jared.ottleys.net/alfresco/tunneling-debug-and-jmx-for-alfresco The -Djava.rmi.server.hostname=dummyhost option is needed to help RMI know where to connect. RMI connects in a two part process. First by connecting to the RMI server registry, which pushes your request to the JMX service which is dynamically allocated on the first open port available to it at start up time. On Aug 30, 2010, at 10:30 AM, Juho Mäkinen wrote: I think that JMX needs additional ports to function correctly. Try to disable all firewalls between the client and the server so that client can connect to any port in the server and try again. - Juho Mäkinen On Mon, Aug 30, 2010 at 7:07 PM, Allan Carroll alla...@gmail.com wrote: Hi, I'm trying to manage my cassandra cluster from a remote box and having issues getting nodetool to connect. All the machines I'm using are running on AWS. Here's what happens when I try: /opt/apache-cassandra-0.6.4/bin/nodetool -h xxx.xxx.xxx.143 -p 10036 ring Error connecting to remote JMX agent! java.rmi.ConnectException: Connection refused to host: xxx.xxx.xxx.143; nested exception is: java.net.ConnectException: Connection timed out When I'm local to a box (Ubuntu 10.04) running Cassandra, I can connect fine via both 127.0.0.1 and external ip (xxx.xxx.xxx.143). I can telnet into the jmx port from an external machine fine: telnet xxx.xxx.xxx.143 10036 Trying xxx.xxx.xxx.143... Connected to xxx.xxx.xxx.143. Escape character is '^]'. I already added the -Djava.rmi.server.hostname parameter to the java runtime, but it didn't seem to affect anything. /usr/bin/jsvc -home /usr/lib/jvm/java-6-openjdk/jre -pidfile /var/run/cassandra.pid -errfile 1 -outfile /var/log/cassandra/output.log -cp /usr/share/cassandra/antlr-3.1.3.jar:/usr/share/cassandra/apache-cassandra-0.6.3.jar:/usr/share/cassandra/avro-1.2.0-dev.jar:/usr/share/cassandra/clhm-production.jar:/usr/share/cassandra/commons-cli-1.1.jar:/usr/share/cassandra/commons-codec-1.2.jar:/usr/share/cassandra/commons-collections-3.2.1.jar:/usr/share/cassandra/commons-lang-2.4.jar:/usr/share/cassandra/google-collections-1.0.jar:/usr/share/cassandra/hadoop-core-0.20.1.jar:/usr/share/cassandra/high-scale-lib.jar:/usr/share/cassandra/ivy-2.1.0.jar:/usr/share/cassandra/jackson-core-asl-1.4.0.jar:/usr/share/cassandra/jackson-mapper-asl-1.4.0.jar:/usr/share/cassandra/jline-0.9.94.jar:/usr/share/cassandra/json-simple-1.1.jar:/usr/share/cassandra/libthrift-r917130.jar:/usr/share/cassandra/log4j-1.2.14.jar:/usr/share/cassandra/slf4j-api-1.5.8.jar:/usr/share/cassandra/slf4j-log4j12-1.5.8.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar -Xmx4G -Xms128M -Djava.rmi.server.hostname=xxx.xxx.xxx.143 -Dcassandra -Dstorage-config=/etc/cassandra -Dcom.sun.management.jmxremote.port=10036 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false org.apache.cassandra.thrift.CassandraDaemon netstat shows that I'm still bound to IP6 netstat -nap|grep 10036 tcp6 0 0 :::10036:::*LISTEN 29277/jsvc And, now I'm at an impasse. Any help would be greatly appreciated. Thanks -Allan
