Hello, Initially we had our setup with both ZK and Kafka on the same nodes as well. Over time though this proved problematic as we both increased the usage of Kafka and had to scale out the cluster. ZK specifically can be very touchy if it gets IO bound - so when the Kafka cluster would get under high load we would start to see slow response and in the worst case scenario a cascading failure. Also, from my understanding ZK doesn't really scale well past five nodes - so once your cluster grows large enough your nodes become non-homogeneous as you're only running ZK on certain nodes. So from our experience it was much cleaner to spin up a 3 node ZK cluster for each Kafka cluster we ran, dedicated to that cluster. That allows us to grow out the Kafka cluster when needed without ever worrying about ZK. In most cases we're more concerned about the ZK integrity and keeping that solid and find it fairly easy to reprovision/add Kafka nodes when necessary. Hope that helps.
Craig On Mon, Sep 19, 2016 at 3:01 PM, Digumarthi, Prabhakar Venkata Surya < prabhakarvenkatasurya.digumar...@capitalone.com> wrote: > Hi Team, > > > What are the downsides of installing Zookeeper and kafka on same machine, > in multi broker environment? > > We are trying to install Zookeeper and kafka in AWS world and its becoming > difficult for us to maintain ZK and Kafka with some issues. Also > re-provisioning ZK and Kafka instances separately is getting complicated. > Thanks, > Prabhakar > ________________________________________________________ > > The information contained in this e-mail is confidential and/or > proprietary to Capital One and/or its affiliates and may only be used > solely in performance of work or services for Capital One. The information > transmitted herewith is intended only for use by the individual or entity > to which it is addressed. If the reader of this message is not the intended > recipient, you are hereby notified that any review, retransmission, > dissemination, distribution, copying or other use of, or taking of any > action in reliance upon this information is strictly prohibited. If you > have received this communication in error, please contact the sender and > delete the material from your computer. >
