Hi again, I have a lot of logs. Would someone like to collaborate on this strange situation?
I feel like it's very interesting. We have been missing several partitions for a month. Which is very strange for fault-tolerated products)). I'm sure that I have some missing parts from this puzzle that's why I need to discuss this. Best regards, Bulanova Daria On Mon, Sep 12, 2022 at 6:39 PM Daria Bulanova < daria.bulan...@hellofresh.com> wrote: > Hi all, > > I've faced a very strange situation when rebalancing happens. We have a > kafka-connect cluster that manages around 20 Aws S3 sink connectors. Each > connector has between 3-10 tasks. All these tasks are handled with 6 kube > pods and lands data to S3. > > A week ago we found that we missed data from the topic with the highest > rate. Deep analytics showed that we missed data from whole partitions when > the rebalancing round was triggered. > > I went through the logs and found out that the rebalancing round assigned > empty partitions for several tasks instead of assigning the real one. > > This particular topic has 10 partitions and the situation looks like this: > > https://drive.google.com/file/d/1XYZJH8vlk-cCsuFXQ890sZj-hDG12SNA/view?usp=sharing > > The first question is why does kafka-connect work without certain > partitions 0,3,5 as you can see on the most left assignment column on the > picture? > > I feel like all this happens due to what I would call a rebalancing storm. > Because we usually are in the deployment process we have around 12-15 > rounds of rebalancing. Also, we found out that it happens with a topic that > has the most input rate. but I don't understand why kafka-connect works > very silently in this case. I mean without any exception or something like > this. > > Could someone navigate me? > Thank you in advance. > > Best regards, > Bulanova Daria > -- <https://www.hellofresh.com/jobs/?utm_medium=email&utm_source=email_signature> HelloFresh SE, Berlin (Sitz der Gesellschaft) | Vorstände: Dominik S. Richter (Vorsitzender), Thomas W. Griesel, Christian Gärtner, Edward Boyes | Vorsitzender des Aufsichtsrats: John H. Rittenhouse | Eingetragen beim Amtsgericht Charlottenburg, HRB 182382 B | USt-Id Nr.: DE 302210417 *CONFIDENTIALITY NOTICE:* This message (including any attachments) is confidential and may be privileged. It may be read, copied and used only by the intended recipient. If you have received it in error please contact the sender (by return e-mail) immediately and delete this message. Any unauthorized use or dissemination of this message in whole or in parts is strictly prohibited.
