On May 26, 2015, at 1:44 PM, Joel Koshy <jjkosh...@gmail.com> wrote:
>> Apologies if this question has been asked before. If I understand things >> correctly a client can only fetch from the leader of a partition, not from >> an (in-sync) replica. I have a use case where it would be very beneficial >> if it were possible to fetch from a replica instead of just the leader, and >> I wonder why it is not allowed? Are there any consistency problems with >> allowing it, for example? Is there any way to configure Kafka to allow it? > > Yes this should be possible. I don't think there are any consistency > issues (barring any bugs) since we never expose past the > high-watermark and the follower HW is strictly <= leader HW. Can you > file a jira for this? > Wouldn't this allow Kafka to scale to handle a lot more consumer traffic? Currently, consumers all have to read from the leader, which means that the network/disk bandwidth of a particular leader is the bottleneck. If consumers could read from in-sync replicas, then a single node no longer is the bottleneck for reads. You could scale out your read capacity as far as you want. -James >> The use case is a Kafka cluster running in EC2 across three availability >> zones. > > Out of curiosity - what's the typical latency (distribution) you see > between zones? > > Joel
