From: Guozhang Wang wangg...@gmail.com
Sent: Monday, February 9, 2015 10:38 AM
To: users@kafka.apache.org
Subject: Re: Poor performance consuming multiple topics
Hello
...@gmail.com
Sent: Monday, February 9, 2015 10:38 AM
To: users@kafka.apache.org
Subject: Re: Poor performance consuming multiple topics
Hello CJ,
You have to set the fetch size to be = the maximum message size possible,
otherwise the consumption will block upon encountering these large
messages
Hello CJ,
You have to set the fetch size to be = the maximum message size possible,
otherwise the consumption will block upon encountering these large messages.
I am wondering by saying poor performance what do you mean exactly? Are
you seeing low throughput, and can you share your consumer
From: Guozhang Wang wangg...@gmail.com
Sent: Monday, February 9, 2015 10:38 AM
To: users@kafka.apache.org
Subject: Re: Poor performance consuming multiple topics
Hello CJ,
You have
Hi CJ,
I recently ran into some kafka message size related issue and did some
digging around to understand the system. I will put those details in brief
and hope it will help you.
Each consumer connector has fetcher threads and fetcher manager threads
associated with it. The Fetcher thread talks
Hi Kafka team,
We have a use case where we need to consume from ~20 topics (each with 24
partitions), we have a potential max message size of 20MB so we've set our
consumer fetch.size to 20MB but that's causing very poor performance on our
consumer (most of our messages are in the 10-100k
