Bryan, This sounds related to https://cwiki.apache.org/confluence/display/KAFKA/KIP-134%3A+Delay+initial+consumer+group+rebalance and https://issues.apache.org/jira/browse/KAFKA-4925.
-James > On Jun 13, 2017, at 7:02 AM, Bryan Baugher <bjb...@gmail.com> wrote: > > The topics already exist prior to starting any of the consumers > > On Mon, Jun 12, 2017 at 9:35 PM J Pai <jai.forums2...@gmail.com> wrote: > >> When are the topics on which these consumer groups consume, created? >> >> -Jaikiran >> On 13-Jun-2017, at 3:18 AM, Bryan Baugher <bjb...@gmail.com> wrote: >> >> Hi everyone, >> >> We are currently experiencing slow startup times for our consumer groups >> (16-32 processes for a hundred or more partitions) in the range of minutes >> (3-15 minutes), where little to no messages are consumed before suddenly >> everything just starts working at full speed. >> >> I'm currently using Kafka 0.9.0.1 but we are in the middle of upgrading to >> Kafka 0.10.2.1. We also using the newer kafka consumer API and group >> management on a simple Apache Storm topology. We don't make use of Storm's >> kafka spout but instead wrote a simple one ourselves. >> >> Using the kafka AdminClient I was able to poll for consumer group summary >> information. What I've found is that the group seems to sit >> in PreparingRebalance state for minutes before finally becoming Stable >> which then everything starts processing quickly. I've also enabled debug >> logging around the consumer's coordinator classes but didn't see anything >> to indicate the issue. >> >> I'm hoping that just upgrading to 0.10 or tweaking how we use our consumer >> in Apache Storm is the problem but are there any pointers on things I >> should look at or try? >> >> Bryan >> >>
