8 hours, 1 cup of coffee, and 4 Advil later, and I think I got the bottom of this. Not having much of a Java or JMX background, I'll try to explain it the best that I can.
To recap, my machine originally had the IP address of 10.244.207.16. Then I shutdown/restarted that EC2 instance, and it had the IP 10.84.117.110. During this, Cassandra was fine -- I could still connect to 127.0.0.1 with cqlsh and the Helenus node.js module. Things got weird only when I tried to use nodetool to manage the instance. As best I can tell, here's the algorithm that nodetool uses when connecting to a Cassandra instance: Step 1) Connect to the hostname and port specified on the command line. "localhost" and "7199" are the defaults. Step 2) Cassandra, at boot time, notes the hostname of the machine, and tells nodetool "go connect to this hostname instead!" After further investigation, it seems that after my instance was rebooted, the file /etc/hostname was not updated. It still had the value "ip-10-244-207-16.ec2.internal" in it. This means that any attempt to connect to Cassandra involved Cassandra telling nodetool, "Hey, go talk to 10.244.207.16 instead". And that's where things went wrong. The permanent fix for this was to change the hostname to "localhost" and to restart Cassandra. The fact that Cassandra notes the hostname at startup was one thing that made this so difficult to track down. I did not see the old IP anywhere in Cassandra configuration (or in logfile output), so I did not think there was anything abnormal happening in the instance. While I'm sure there's a good reason for this sort of behavior, it is very confusing to a Cassandra newbie such as myself, and I'll bet others have been affected by this as well. In the future, I think some sort of logging of this sort of of logic, or perhaps a --verbose mode for nodetool would be a really good idea. What do other folks think? -- Doug http://twitter.com/dmuth On Tue, May 29, 2012 at 12:08 PM, Douglas Muth <doug.m...@gmail.com> wrote: > I'm afraid that did not work. I'm running JMX on port 7199 (the > default) and I verified that the port is open and accepting > connections. [snip]
