I'll try to push for an upgrade, but I'd like to at least understand what's going on here. Am I the only one that don't understand why the rebalancing is happenning looking at the logs? ________________________________ De : Luke Chen <[email protected]> Envoyé : vendredi 3 décembre 2021 04:48 À : Kafka Users <[email protected]> Objet : Re: Rebalancing taking more than 15/20 minutes to complete
CAUTION: EXTERNAL EMAIL. Do not click links or open unless you recognize the sender and know the content is safe. Hi Michael, I'm not 100% sure if you've hit the issue on KAFKA-9752 <https://issues.apache.org/jira/browse/KAFKA-9752>. >From the description in the KAFKA-9752 <https://issues.apache.org/jira/browse/KAFKA-9752>, you are: 1) we have a consumer using an old JoinGroup version, 2) the consumer times out and disconnects from its initial JoinGroup request. It's just not sure if your consumer group is stuck in `PreparingRebalance` state or not. But since the v2.3.1 is quite old and we know it can be fixed by upgrading to the newer version, I would recommend to upgrade to more recent version. Thank you. Luke On Fri, Dec 3, 2021 at 2:48 AM Michael Longo <[email protected]> wrote: > Hello, > I'm having an issue with one of our application and I just can't find > what's going on. We've got 9 instances of the said application, running the > exact same base code, and sometimes (often) when we restart one of the > application we find ourselves in a rebalancing nightmare for several > minutes, with decreased performance on the system in the mean time. > > We've got 3 kafka deployed running version 2.12-2.3.1 and we're using the > kafkastream api version 2.7.0 with Java. > > I've found https://issues.apache.org/jira/browse/KAFKA-9752 that could > apply to our broker's version but I'm not really sure how to know for sure > if that's our issue or not. > > I've enabled org.apache.kafka logger as TRACE and got a lot of logs that I > tried to go through, but I just couldn't find the cause of the rebalance. > > If anyone wants to have a look, here are our logs for a given minute when > we've had the issue. > Note: I've restricted to one minute as it seemed enough, but if needed I > can provide more > Note2: I had the issue with 2 consumers here, > 'staging-message-processor-retry2' and 'staging-message-processor-retry3' > Note3: One thing that might cause the issue, but I've not really been able > to confirm/deny it yet, is that we've got some sort of custom throttling > going on where we put the consumption threads to sleep sometime for a few ms > > Unfiltered server's logs: > https://stuff.stooit.com/d/1/61a912cd2cad0/kafkaserver-3.txt > https://stuff.stooit.com/d/1/61a912cd1418e/kafkaserver-5.txt > > Filtered app's logs: > Filtered with: grep "02.12.2021 17:25:" | grep -v "some consumer groups > we don't care about" | grep -vi producer | perl -p -e 's/some names or > domains/redacted/' > <https://stuff.stooit.com/d/1/61a9108bb1a0b/kafkaapp-1.log> > https://stuff.stooit.com/d/1/61a912ce06ddd/kafkaapp-1.txt > https://stuff.stooit.com/d/1/61a912cde275b/kafkaapp-2.txt > https://stuff.stooit.com/d/1/61a912cdd21fc/kafkaapp-3.txt > https://stuff.stooit.com/d/1/61a912cdb78e6/kafkaapp-4.txt > https://stuff.stooit.com/d/1/61a912cd9f161/kafkaapp-5.txt > https://stuff.stooit.com/d/1/61a912cd7a5bb/kafkaapp-6.txt > https://stuff.stooit.com/d/1/61a912cd61db2/kafkaapp-7.txt > https://stuff.stooit.com/d/1/61a912cd5385f/kafkaapp-8.txt > https://stuff.stooit.com/d/1/61a912cd4535e/kafkaapp-9.txt > > Thanks, don't hesitate if you need more details > Michael > > Your privacy is important to us. Please see our Privacy Notice< > https://www.sigfox.com/en/privacy-and-cookies-policy> for further > details. The information contained in this Message is confidential. If you > are not the addressee, you may not copy, forward, disclose or use any part > of it. If you have received this Message in error, please delete it and all > copies from your system and notify the sender immediately by return > message. Any use of information contained in this Message not in accordance > with its intended purpose, any dissemination or disclosure (either whole or > partial), is prohibited unless expressly authorized. Email communication > cannot be guaranteed to be timely secure, error or virus-free. The sender > cannot be held responsible for any alteration, errors or omissions, which > arise as a result. > > > .................................................................................................................. > > La protection de vos données personnelles est primordiale pour notre > établissement. Merci de consulter notre notice sur la protection des > données personnelles <https://www.sigfox.com/en/privacy-and-cookies-policy> > pour plus d'informations. Ce message et toutes les pièces jointes (ci-après > le 'Message') sont établis à l'intention exclusive des destinataires. Les > informations qui y figurent sont confidentielles. Si vous n'êtes pas le > destinataire de ce Message, il vous est interdit de le copier, de le faire > suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez reçu > ce Message par erreur, merci de le supprimer de votre système, ainsi que > toutes ses copies, et de n'en garder aucune trace sur quelque support que > ce soit. Veuillez également en avertir immédiatement l'expéditeur par > retour du Message. Toute utilisation de ce Message non conforme à sa > destination, toute diffusion ou toute publication totale ou partielle, est > interdite sauf autorisation expresse. Il est impossible de garantir que les > communications par messagerie électronique arrivent en temps utile, soient > sécurisées ou dénuées de toute erreur ou virus. L'expéditeur ne peut être > tenu responsable des modifications, erreurs ou omissions qui pourraient en > résulter. > Your privacy is important to us. Please see our Privacy Notice<https://www.sigfox.com/en/privacy-and-cookies-policy> for further details. The information contained in this Message is confidential. If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this Message in error, please delete it and all copies from your system and notify the sender immediately by return message. Any use of information contained in this Message not in accordance with its intended purpose, any dissemination or disclosure (either whole or partial), is prohibited unless expressly authorized. Email communication cannot be guaranteed to be timely secure, error or virus-free. The sender cannot be held responsible for any alteration, errors or omissions, which arise as a result. .................................................................................................................. La protection de vos données personnelles est primordiale pour notre établissement. Merci de consulter notre notice sur la protection des données personnelles <https://www.sigfox.com/en/privacy-and-cookies-policy> pour plus d'informations. Ce message et toutes les pièces jointes (ci-après le 'Message') sont établis à l'intention exclusive des destinataires. Les informations qui y figurent sont confidentielles. Si vous n'êtes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez reçu ce Message par erreur, merci de le supprimer de votre système, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Veuillez également en avertir immédiatement l'expéditeur par retour du Message. Toute utilisation de ce Message non conforme à sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. Il est impossible de garantir que les communications par messagerie électronique arrivent en temps utile, soient sécurisées ou dénuées de toute erreur ou virus. L'expéditeur ne peut être tenu responsable des modifications, erreurs ou omissions qui pourraient en résulter.
