Hello, I'm having an issue with one of our application and I just can't find what's going on. We've got 9 instances of the said application, running the exact same base code, and sometimes (often) when we restart one of the application we find ourselves in a rebalancing nightmare for several minutes, with decreased performance on the system in the mean time.
We've got 3 kafka deployed running version 2.12-2.3.1 and we're using the kafkastream api version 2.7.0 with Java. I've found https://issues.apache.org/jira/browse/KAFKA-9752 that could apply to our broker's version but I'm not really sure how to know for sure if that's our issue or not. I've enabled org.apache.kafka logger as TRACE and got a lot of logs that I tried to go through, but I just couldn't find the cause of the rebalance. If anyone wants to have a look, here are our logs for a given minute when we've had the issue. Note: I've restricted to one minute as it seemed enough, but if needed I can provide more Note2: I had the issue with 2 consumers here, 'staging-message-processor-retry2' and 'staging-message-processor-retry3' Note3: One thing that might cause the issue, but I've not really been able to confirm/deny it yet, is that we've got some sort of custom throttling going on where we put the consumption threads to sleep sometime for a few ms Unfiltered server's logs: https://stuff.stooit.com/d/1/61a912cd2cad0/kafkaserver-3.txt https://stuff.stooit.com/d/1/61a912cd1418e/kafkaserver-5.txt Filtered app's logs: Filtered with: grep "02.12.2021 17:25:" | grep -v "some consumer groups we don't care about" | grep -vi producer | perl -p -e 's/some names or domains/redacted/' <https://stuff.stooit.com/d/1/61a9108bb1a0b/kafkaapp-1.log> https://stuff.stooit.com/d/1/61a912ce06ddd/kafkaapp-1.txt https://stuff.stooit.com/d/1/61a912cde275b/kafkaapp-2.txt https://stuff.stooit.com/d/1/61a912cdd21fc/kafkaapp-3.txt https://stuff.stooit.com/d/1/61a912cdb78e6/kafkaapp-4.txt https://stuff.stooit.com/d/1/61a912cd9f161/kafkaapp-5.txt https://stuff.stooit.com/d/1/61a912cd7a5bb/kafkaapp-6.txt https://stuff.stooit.com/d/1/61a912cd61db2/kafkaapp-7.txt https://stuff.stooit.com/d/1/61a912cd5385f/kafkaapp-8.txt https://stuff.stooit.com/d/1/61a912cd4535e/kafkaapp-9.txt Thanks, don't hesitate if you need more details Michael Your privacy is important to us. Please see our Privacy Notice<https://www.sigfox.com/en/privacy-and-cookies-policy> for further details. The information contained in this Message is confidential. If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this Message in error, please delete it and all copies from your system and notify the sender immediately by return message. Any use of information contained in this Message not in accordance with its intended purpose, any dissemination or disclosure (either whole or partial), is prohibited unless expressly authorized. Email communication cannot be guaranteed to be timely secure, error or virus-free. The sender cannot be held responsible for any alteration, errors or omissions, which arise as a result. .................................................................................................................. La protection de vos données personnelles est primordiale pour notre établissement. Merci de consulter notre notice sur la protection des données personnelles <https://www.sigfox.com/en/privacy-and-cookies-policy> pour plus d'informations. Ce message et toutes les pièces jointes (ci-après le 'Message') sont établis à l'intention exclusive des destinataires. Les informations qui y figurent sont confidentielles. Si vous n'êtes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez reçu ce Message par erreur, merci de le supprimer de votre système, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Veuillez également en avertir immédiatement l'expéditeur par retour du Message. Toute utilisation de ce Message non conforme à sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. Il est impossible de garantir que les communications par messagerie électronique arrivent en temps utile, soient sécurisées ou dénuées de toute erreur ou virus. L'expéditeur ne peut être tenu responsable des modifications, erreurs ou omissions qui pourraient en résulter.
