Hello Navneeth, Just to verify some behaviors, could you try 1) not using instance.id config, hence no static members, 2) upgrade to the latest version of Kafka, respectively (i.e. do not do them at the same time) and see if either one of them help with the imbalance issue?
On Sun, Feb 20, 2022 at 2:17 AM Luke Chen <show...@gmail.com> wrote: > Hi Navneeth, > > To know the reason why there's more than one partition in the same stream > task, we should know why the rebalance triggered. > That might have to look into the logs. > > > I have configured standby to be 1 which means there will be > one more copy of the state store and warm up by default is 2. What's the > difference, will there be 2 copies now? > > You should have "at most" 3 copies now, which is 1 standby + 2 warmup when > having enough Kafka stream instances. > > Thank you. > Luke > > On Sat, Feb 19, 2022 at 2:18 PM Navneeth Krishnan < > reachnavnee...@gmail.com> > wrote: > > > Hi Guozhang, > > > > Thanks and sorry for the late reply. I'm overriding the > > GROUP_INSTANCE_ID_CONFIG > > & APPLICATION_SERVER_CONFIG. > > Rest all are defaults. Even then I see more than one partition being > > allocated to the same stream task. > > > > Also I have an additional question regarding the replicas. The default > > values for the configs num.standby.replicas & max.warmup.replicas are 0 > & 2 > > respectively. I have configured standby to be 1 which means there will be > > one more copy of the state store and warm up by default is 2. What's the > > difference, will there be 2 copies now? > > > > Thanks > > > > On Fri, Feb 4, 2022 at 1:13 AM Guozhang Wang <wangg...@gmail.com> wrote: > > > > > Hello Navneeth, > > > > > > Could you describe how you ended up with more than one partition > > > assigned to the same thread after certain rebalance(s)? Do you override > > any > > > default config values such as instance.id (for static consumer > members), > > > etc? > > > > > > Also I'd suggest upgrading to a newer version --- we just released > 3.1.0 > > > --- since we've made many improvements / fix bugs around rebalances and > > > assignment logic since 2.6. > > > > > > > > > Guozhang > > > > > > On Wed, Feb 2, 2022 at 9:37 AM Navneeth Krishnan < > > reachnavnee...@gmail.com > > > > > > > wrote: > > > > > > > Hi All, > > > > > > > > We are facing an issue with our kafka streams application due to > uneven > > > > task allocation. There are 100 partitions in the input topic with 100 > > > > stream threads processing the data. Everything works well when each > > task > > > > gets assigned with 1 partition. But when more than one partition is > > > > assigned to the same thread then it causes delay in processing > causing > > a > > > > huge backlog. > > > > > > > > How is everyone handling this? This creates a huge impact to the > > > > application SLA and we would like to minimize such cases. Any > > suggestions > > > > would be appreciated. > > > > > > > > Note: we are on version 2.6.2 > > > > > > > > Thanks, > > > > Navneeth > > > > > > > > > > > > > -- > > > -- Guozhang > > > > > > -- -- Guozhang
