Hi Michael, I'm not 100% sure if you've hit the issue on KAFKA-9752 <https://issues.apache.org/jira/browse/KAFKA-9752>. >From the description in the KAFKA-9752 <https://issues.apache.org/jira/browse/KAFKA-9752>, you are: 1) we have a consumer using an old JoinGroup version, 2) the consumer times out and disconnects from its initial JoinGroup request.
It's just not sure if your consumer group is stuck in `PreparingRebalance` state or not. But since the v2.3.1 is quite old and we know it can be fixed by upgrading to the newer version, I would recommend to upgrade to more recent version. Thank you. Luke On Fri, Dec 3, 2021 at 2:48 AM Michael Longo <[email protected]> wrote: > Hello, > I'm having an issue with one of our application and I just can't find > what's going on. We've got 9 instances of the said application, running the > exact same base code, and sometimes (often) when we restart one of the > application we find ourselves in a rebalancing nightmare for several > minutes, with decreased performance on the system in the mean time. > > We've got 3 kafka deployed running version 2.12-2.3.1 and we're using the > kafkastream api version 2.7.0 with Java. > > I've found https://issues.apache.org/jira/browse/KAFKA-9752 that could > apply to our broker's version but I'm not really sure how to know for sure > if that's our issue or not. > > I've enabled org.apache.kafka logger as TRACE and got a lot of logs that I > tried to go through, but I just couldn't find the cause of the rebalance. > > If anyone wants to have a look, here are our logs for a given minute when > we've had the issue. > Note: I've restricted to one minute as it seemed enough, but if needed I > can provide more > Note2: I had the issue with 2 consumers here, > 'staging-message-processor-retry2' and 'staging-message-processor-retry3' > Note3: One thing that might cause the issue, but I've not really been able > to confirm/deny it yet, is that we've got some sort of custom throttling > going on where we put the consumption threads to sleep sometime for a few ms > > Unfiltered server's logs: > https://stuff.stooit.com/d/1/61a912cd2cad0/kafkaserver-3.txt > https://stuff.stooit.com/d/1/61a912cd1418e/kafkaserver-5.txt > > Filtered app's logs: > Filtered with: grep "02.12.2021 17:25:" | grep -v "some consumer groups > we don't care about" | grep -vi producer | perl -p -e 's/some names or > domains/redacted/' > <https://stuff.stooit.com/d/1/61a9108bb1a0b/kafkaapp-1.log> > https://stuff.stooit.com/d/1/61a912ce06ddd/kafkaapp-1.txt > https://stuff.stooit.com/d/1/61a912cde275b/kafkaapp-2.txt > https://stuff.stooit.com/d/1/61a912cdd21fc/kafkaapp-3.txt > https://stuff.stooit.com/d/1/61a912cdb78e6/kafkaapp-4.txt > https://stuff.stooit.com/d/1/61a912cd9f161/kafkaapp-5.txt > https://stuff.stooit.com/d/1/61a912cd7a5bb/kafkaapp-6.txt > https://stuff.stooit.com/d/1/61a912cd61db2/kafkaapp-7.txt > https://stuff.stooit.com/d/1/61a912cd5385f/kafkaapp-8.txt > https://stuff.stooit.com/d/1/61a912cd4535e/kafkaapp-9.txt > > Thanks, don't hesitate if you need more details > Michael > > Your privacy is important to us. Please see our Privacy Notice< > https://www.sigfox.com/en/privacy-and-cookies-policy> for further > details. The information contained in this Message is confidential. If you > are not the addressee, you may not copy, forward, disclose or use any part > of it. If you have received this Message in error, please delete it and all > copies from your system and notify the sender immediately by return > message. Any use of information contained in this Message not in accordance > with its intended purpose, any dissemination or disclosure (either whole or > partial), is prohibited unless expressly authorized. Email communication > cannot be guaranteed to be timely secure, error or virus-free. The sender > cannot be held responsible for any alteration, errors or omissions, which > arise as a result. > > > .................................................................................................................. > > La protection de vos données personnelles est primordiale pour notre > établissement. Merci de consulter notre notice sur la protection des > données personnelles <https://www.sigfox.com/en/privacy-and-cookies-policy> > pour plus d'informations. Ce message et toutes les pièces jointes (ci-après > le 'Message') sont établis à l'intention exclusive des destinataires. Les > informations qui y figurent sont confidentielles. Si vous n'êtes pas le > destinataire de ce Message, il vous est interdit de le copier, de le faire > suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez reçu > ce Message par erreur, merci de le supprimer de votre système, ainsi que > toutes ses copies, et de n'en garder aucune trace sur quelque support que > ce soit. Veuillez également en avertir immédiatement l'expéditeur par > retour du Message. Toute utilisation de ce Message non conforme à sa > destination, toute diffusion ou toute publication totale ou partielle, est > interdite sauf autorisation expresse. Il est impossible de garantir que les > communications par messagerie électronique arrivent en temps utile, soient > sécurisées ou dénuées de toute erreur ou virus. L'expéditeur ne peut être > tenu responsable des modifications, erreurs ou omissions qui pourraient en > résulter. >
