Thank you all for your responses. So here is what I tried and it worked since EIP is not an option for me - 1) Created an ENI with a dedicated IP 2) Associated that IP with A Type address 3) Assigned that ENI to the EC2 instance 4) Created EBS volume to keep ZK data
As EBS volume and ENI are bound to AZ, I have created AutoScaling group per AZ. Thanks, Zuber On Wed, Aug 3, 2016 at 10:22 AM, Zuber <objectsp...@gmail.com> wrote: > Hello – > > > > We are planning to use Kafka as Event Store in a system which is being built > using event sourcing design approach. > > Here is how we deployed the cluster in AWS to verify HA in the cloud (in our > POC we only had 1 topic with 1 partition and 3 replication factor) - > > 1) 3 ZK servers running in different AZs (managed by Auto Scaling Group) > > 2) 3 Kafka brokers EC2 running in different AZs (managed by Auto Scaling > Group) > > 3) Kafka logs are stored in EBS volumes > > 4) A type addresses are defined for all ZK servers & Kafka brokers in > Route53 > > EC2 instance registers its IP for corresponding A type address (in Route53) > on startup > > > > But due a bug in ZKClient used by Kafka broker which caches ZK IP forever, I > don’t see any other option other than bouncing all brokers. > > > > One of the Netflix presentation (following links) mentions about the issue > as well as couple of ZK JIRA defects but I haven’t found any concrete > solution yet. > > I would really appreciate any help in this regard. > > > > http://image.slidesharecdn.com/netflix-kafka-150325105558-conversion-gate01/95/netflix-data-pipeline-with-kafka-36-638.jpg?cb=1427281139 > > http://image.slidesharecdn.com/netflix-kafka-150325105558-conversion-gate01/95/netflix-data-pipeline-with-kafka-36-638.jpg?cb=1427281139 > > https://issues.apache.org/jira/browse/ZOOKEEPER-338 > > https://issues.apache.org/jira/browse/ZOOKEEPER-1506 > > http://grokbase.com/t/kafka/users/131x67h1bt/zookeeper-caching-dns-entries > > > > Thanks, > > Zuber > > -- Thanks & Regards, Zuber Saiyed
