Hello Tony, I think you already know the consumer-client side fetch-from-follower feature: https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica
Say your Kafka deployment is cross-AZs, and your Streams is also deployed cross-AZ, you can, by setting the "client.rack" through streams configs, to let the embedded consumers to read from close replicas within the same AZ as well. Guozhang On Mon, Oct 19, 2020 at 9:19 PM Tony John <tonyjohnant...@gmail.com> wrote: > Thanks Matthias.. I don't think I will be able to take it up.. Will wait > for it to be available in future. :) > > On Mon, Oct 19, 2020 at 11:56 PM Matthias J. Sax <mj...@apache.org> wrote: > > > No suggestions... Sorry. > > > > The task assignment algorithm cannot be customized. > > > > The only thing you _could_ do, is to pickup the ticket yourself (to get > > the feature maybe in 2.8 release). Not sure if you would be interested > > to contribute :) > > > > > > -Matthias > > > > On 10/19/20 11:08 AM, Tony John wrote: > > > Thanks for the quick response Matthias. We were planning to move to AWS > > > MSK, and I think we will have to defer it until rack awareness is in > > place > > > for Kafka Streams. Reason being consumers consuming from brokers in > > > different availability zones would result in high data transfer > charges. > > Do > > > you have any suggestions? > > > > > > Thanks, > > > Tony > > > > > > On Mon, Oct 19, 2020 at 11:08 PM Matthias J. Sax <mj...@apache.org> > > wrote: > > > > > >> There is no such feature for Kafka Streams yet. > > >> > > >> We have one ticket for rack aware standby task placement > > >> (https://issues.apache.org/jira/browse/KAFKA-6718) but nobody is > > working > > >> on it. > > >> > > >> > > >> -Matthias > > >> > > >> On 10/19/20 9:20 AM, Tony John wrote: > > >>> Hi All, > > >>> > > >>> I have been trying to figure out some documentation on how to enable > > rack > > >>> awareness on a Kafka Streams app. I do see the broker.rack > > configuration > > >>> which needs to be done at the broker side and client.rack > configuration > > >> for > > >>> the consumers. Is there any specific configuration which is required > to > > >>> enable rack awareness in a streams app? Please throw some light on > > this. > > >>> > > >>> PS: I am using Kafka / Kafka Streams 2.2.1 > > >>> > > >>> Thanks, > > >>> Tony > > >>> > > >> > > > > > > -- -- Guozhang
