While I do have some logs its not trivial to share since the logs are across 16 JVMs and a few different hosts.
On Wed, Jun 14, 2017 at 10:34 AM Eno Thereska <eno.there...@gmail.com> wrote: > The delay in that KIP is just 3 seconds, not minutes though, right? Would > you have any logs to share? > > Thanks > Eno > > On 14 Jun 2017, at 16:14, Bryan Baugher <bjb...@gmail.com> wrote: > > > > Our consumer group isn't doing anything stateful and we've seen this > > behavior for existing groups as well. It seems like timing could be an > > issue, thanks for the information. > > > > On Tue, Jun 13, 2017 at 7:39 PM James Cheng <wushuja...@gmail.com> > wrote: > > > >> Bryan, > >> > >> This sounds related to > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-134%3A+Delay+initial+consumer+group+rebalance > >> and https://issues.apache.org/jira/browse/KAFKA-4925. > >> > >> -James > >> > >>> On Jun 13, 2017, at 7:02 AM, Bryan Baugher <bjb...@gmail.com> wrote: > >>> > >>> The topics already exist prior to starting any of the consumers > >>> > >>> On Mon, Jun 12, 2017 at 9:35 PM J Pai <jai.forums2...@gmail.com> > wrote: > >>> > >>>> When are the topics on which these consumer groups consume, created? > >>>> > >>>> -Jaikiran > >>>> On 13-Jun-2017, at 3:18 AM, Bryan Baugher <bjb...@gmail.com> wrote: > >>>> > >>>> Hi everyone, > >>>> > >>>> We are currently experiencing slow startup times for our consumer > groups > >>>> (16-32 processes for a hundred or more partitions) in the range of > >> minutes > >>>> (3-15 minutes), where little to no messages are consumed before > suddenly > >>>> everything just starts working at full speed. > >>>> > >>>> I'm currently using Kafka 0.9.0.1 but we are in the middle of > upgrading > >> to > >>>> Kafka 0.10.2.1. We also using the newer kafka consumer API and group > >>>> management on a simple Apache Storm topology. We don't make use of > >> Storm's > >>>> kafka spout but instead wrote a simple one ourselves. > >>>> > >>>> Using the kafka AdminClient I was able to poll for consumer group > >> summary > >>>> information. What I've found is that the group seems to sit > >>>> in PreparingRebalance state for minutes before finally becoming Stable > >>>> which then everything starts processing quickly. I've also enabled > debug > >>>> logging around the consumer's coordinator classes but didn't see > >> anything > >>>> to indicate the issue. > >>>> > >>>> I'm hoping that just upgrading to 0.10 or tweaking how we use our > >> consumer > >>>> in Apache Storm is the problem but are there any pointers on things I > >>>> should look at or try? > >>>> > >>>> Bryan > >>>> > >>>> > >> > >> > >
