Hi Ben, Yes we have a VIP fronting our kafka 0.7 clusters. The producers connect with a broker list that just points to the VIP.
In 0.8, with the new producer at least, the producer doesn't go through Zookeeper - which is good. However, all producers will connect to all brokers to send messages. My concern is that the brokers won't be able to handle the load of 100k+ connections. So i want to know if anyone has experience with this sort of load? Will it work? Do i need to take a multi-layered approach to Kafka? i.e., have many front-end kafka clusters that subsets of producers connect to and then mirror data from there to one or more back-end clusters? I'd rather not have to do this as it becomes painful to track data though the system. Any other recommendations on how to go about it? Thanks, Damian On 26 August 2015 at 10:57, Ben Stopford <b...@confluent.io> wrote: > Hi Damian > > Just clarifying - you’re saying you currently have Kafka 0.7.x running > with a dedicated broker addresses (bypassing ZK) and hitting a VIP which > you use for load balancing writes. Is that correct? > > Are you worried about something specific in the 0.8.x way of doing things > (ZK under that level of load etc) or are you just looking for experiences > running 0.8.x with that many producers? > > B > > > > On 25 Aug 2015, at 10:29, Damian Guy <damian....@gmail.com> wrote: > > > > Hi, > > > > We currently run 0.7.x on our clusters and are now finally getting around > > to upgrading to kafka latest. One thing that has been holding us back is > > that we can no longer use a VIP to front the clusters. I understand we > > could use a VIP for metadata lookups, but we have 100,000 + producers to > at > > least one of our clusters. > > So, my question is: How is Kafka 0.8.x going to handle 100,000+ > producers? > > Any recommendations on setup etc? > > > > Thanks for the help, > > Damian > >
