Hi Liam,
(+adding imgur links to the images)
First of all, thanks for checking my doubt.
I understand that the reason I notice this behavior is because our case
differs from yours in one point: I'm not writing terabytes in a single day.
I'm writing MB, but distribution matters in these MBs, this
Hi, I can't see your images, they're not making it through the mailing
list, sorry.
On Tue, 2 Jun. 2020, 11:09 pm Vinicius Scheidegger, <
vinicius.scheideg...@gmail.com> wrote:
> Hi Liam,
>
> First of all, thanks for checking my doubt.
> I understand that the reason I notice this behavior is
Hi Liam,
First of all, thanks for checking my doubt.
I understand that the reason I notice this behavior is because our case
differs from yours in one point: I'm not writing terabytes in a single day.
I'm writing MB, but distribution matters in these MBs, this
because processing in the consumer
Hi Vinicius,
As you note, the cluster doesn't load balance producers, it relies on them
using a partition strategy to do so.
In production, I've never had actual broker load skew develop from multiple
independent producers using round robining - and we're talking say 20 - 50
producers (depending
Hey guys, I need some help here...
Is this a flaw in the design (maybe a discussion point for a KIP?), is
Kafka not supposed to perform equal load balancing with multiple producers
or am I missing something (which is what I believe is happening)?
On Wed, May 27, 2020 at 2:40 PM Vinicius
Does anyone know whether we could really have an "out of the box" solution
to do round robin over the partitions when we have multiple producers?
By that I mean, a round robin on the broker side (or maybe some way to
synchronize all producers).
Thank you,
On Tue, May 26, 2020 at 1:41 PM Vinicius
Yes, I checked it. The issue is that RoundRobbinPartitioner is bound to the
producer. In a scenario with multiple producers it doesn't guarantee equal
distribution - from what I understood and from my tests, the following
situation happens with it:
[image: image.png]
Of course, the first
Hey Vinicius,
On Tue, 26 May 2020 at 10:27, Vinicius Scheidegger <
vinicius.scheideg...@gmail.com> wrote:
> In a scenario with multiple independent producers (imagine ephemeral
> dockers, that do not know the state of each other), what should be the
> approach for the messages being sent to be
