Hello, I'm looking to convert our recently upgraded Cassandra cluster from a single token per node to using vnodes. We've determined that based on our data consistency and usage patterns that shuffling will be the best way to convert our live cluster.
However, when following the instructions for doing the shuffle, we aren't able to enable shuffling on the other 4 nodes in the cluster. We get the error message 'Failed to enable shuffling', which looks to be a generic string printed when a JMX IOException is caught. Unfortunately, the underlying error is not printed so I'm effectively troubleshooting in the dark. I've done some mailing list diving, as well as Google skimming, and all the suggestions did not seem to work. I've confirmed that a firewall is not the cause as I am able to establish a TCP socket (using telnet) from one node to the other. I've also double-checked the JMX-specific settings that are being set for Cassandra and those look good. I'm going with the most open settings now to try and get this working: -Dcom.sun.management.jmxremote.port=7199 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false I also tried playing with the 'java.rmi.server.hostname' setting, but none of the options set seemed to make a difference (hostname, fqdn, public IPv4 address, private EC2 address). Without any further information from the 'cassandra-shuffle' utility I'm pretty much out of ideas. Any suggestions would be greatly appreciated! Cheers! -Tim
