We've started using an ensemble of 3 zookeepers in an EC2 environment to support a solr cloud setup. I was wondering if anybody else has done so and if they have any recommendations, especially pertaining to instance sizes, DNS, etc.
One particular question I have with regards to EC2: We had an issue a while back where we had to reboot one of our zookeeper instances. Given the way AWS works it meant the server was assigned a new IP address. Our current configuration (generated automatically by an in-house tool) uses the IP addresses of each server in the zookeeper configuration file. That led to issues since the only way we could get the ensemble to recognize the rebooted server was to restart all the zookeepers so that they got the new IP of the rebooted instance. Given the nature of AWS and the likelihood that we'll need to reboot another zookeeper server eventually, what is the best approach to defining the hosts? Can local /etc/hosts files where the we can update the IP be enough, or would we need to switch to using DNS for the zookeepers? I'm guessing we might have to go the DNS route but I'm not 100% convinced. Thanks, -Bruce
