Looking at the code it appears it shouldn't matter what we set the yaml params to. The Ec2MultiRegionSnitch should be using the aws metadata 169.254.169.254 to pick up the internal/external ips as needed.
I think I'll just have to dig in to the code differences between 2.1 and 2.2. We don't want to specify the glboal IP in any of the yaml fields because the global IP for the instance changes if we do an aws instance restart. Don't want yaml editing to be a part of the instance restart process. And I was misinformed, an instance restart in our 2.2 cluster does overwrite the manual system.peers entries, which I expected to happen. On Tue, Mar 26, 2019 at 3:33 AM Oleksandr Shulgin < oleksandr.shul...@zalando.de> wrote: > On Mon, Mar 25, 2019 at 11:13 PM Carl Mueller > <carl.muel...@smartthings.com.invalid> wrote: > >> >> Since the internal IPs are given when the client app connects to the >> cluster, the client app cannot communicate with other nodes in other >> datacenters. >> > > Why should it? The client should only connect to its local data center > and leave communication with remote DCs to the query coordinator. > > >> They seem to be able to communicate within its own datacenter of the >> initial connection. >> > > Did you configure address translation on the client? See: > https://docs.datastax.com/en/developer/java-driver/3.0/manual/address_resolution/#ec2-multi-region > > It appears we fixed this by manually updating the system.peers table's >> rpc_address column back to the public IP. This appears to survive a restart >> of the cassandra nodes without being switched back to private IPs. >> > > I don't think updating system tables is a supported solution. I'm > surprised that even doesn't give you an error. > > Our cassandra.yaml (these parameters are the same in our confs for 2.1 and >> 2.2) has: >> >> listen_address: internal aws vpc ip >> rpc_address: 0.0.0.0 >> broadcast_rpc_address: internal aws vpc ip >> > > It is not straightforward to find the docs for version 2.x anymore, but at > least for 3.0 it is documented that you should set broadcast_rpc_address to > the public IP: > https://docs.datastax.com/en/cassandra/3.0/cassandra/architecture/archSnitchEC2MultiRegion.html > > Regards, > -- > Alex > >
