Cassandra makes the totally reasonable assumption that the entire cluster is in one routable address space. We unfortunately had a situation where: * nodes can talk to each other in the same dc on an internal address, but not talk to each other over their external 1:1 NAT address. * nodes can talk to nodes in the other dc over the external address, but there is no usable shared internal address space they can talk over
In case anyone else finds themselves in the same situation we have what we think is a working solution in pre-production. CASSANDRA-5630 handles the "reconnect trick" to prefer the local ip when in the same DC. And some iptables rules allow the local nodes to do the initial gossiping with each other before that switch. for each node in same dc: 'iptables -t nat -A OUTPUT -j DNAT -p tcp --dst %s --dport 7000 -o eth0 --to-destination %s' % (ext_ip, local_ip)