Re: Broker side partition round robin

[Vinicius Scheidegger]

Hi Liam,

(+adding imgur links to the images)
First of all, thanks for checking my doubt.
I understand that the reason I notice this behavior is because our case
differs from yours in one point: I'm not writing terabytes in a single day.
I'm writing MB, but distribution matters in these MBs, this
because processing in the consumer side is CPU intensive (complex machine
learning algo), so a real equally distributed load is a requirement,
otherwise messages start queuing (which is what I don't want).

Our system is also in production and this bad distribution is generating
queuing in some consumers while others are idle.
I tried both approaches:
Round Robbin - actually gives me a distributed load in a big time window,
but as I grow the number of producers I get several messages in a single
partition while others don't get anything.
Hash of the key - here we used a random UUID as the key - still a bad
distribution


[image: image.png]
(imgur link: https://i.imgur.com/ZhQq9Uh.png )


When I start, let's say 10 producers, we can get 5 messages going to one
partition while others have none - i understand that this is because the
round robin is internal to the producer.
[image: image.png]
  (imgur link: https://i.imgur.com/Hv8TUDL.png 
)
(The picture above is what I believe is happening)


it would surprise me that this hasn't come up before, that's why I'm pretty
sure I'm missing something here...
We're currently analyzing some solutions, one of them is building our own
partitioner with shared memory (yes, that's how far we got on this),
although I believe a better solution would be to have this on Kafka broker
side and not depend on custom code.

[image: image.png]
  (imgur link: https://i.imgur.com/JR8QvZH.png 
)

Above is the draft of one of our current ideas of a possible design. Based
on the shared memory we could decide the partition and send the messages
directly there (the number of producers, partitions and consumers here are
simplified).
This if we don't find a solution within Kafka to mitigate this distribution
issue - even though the thought design imposes limits and add layers we
didn't had in our initial design.

My question is, do we really need to develop this??
Is equal distribution in a scenario with multiple producers something that
can be achieved in Kafka without custom development?
Having never checked out the broker code when receiving a message - is
partition definition even possible on the broker side?
If this really does not exist, would a feature like that benefit other
people, is it worth checking out instead of the above solution?
Should I move this message to the dev forum? - no one gave me much
attention there too (but maybe my messages are too big/boring - hahaha)

Thanks


On Tue, Jun 2, 2020 at 1:08 PM Vinicius Scheidegger <
vinicius.scheideg...@gmail.com> wrote:

> Hi Liam,
>
> First of all, thanks for checking my doubt.
> I understand that the reason I notice this behavior is because our case
> differs from yours in one point: I'm not writing terabytes in a single day.
> I'm writing MB, but distribution matters in these MBs, this
> because processing in the consumer side is CPU intensive (complex machine
> learning algo), so a real equally distributed load is a requirement,
> otherwise messages start queuing (which is what I don't want).
>
> Our system is also in production and this bad distribution is generating
> queuing in some consumers while others are idle.
> I tried both approaches:
> Round Robbin - actually gives me a distributed load in a big time window,
> but as I grow the number of producers I get several messages in a single
> partition while others don't get anything.
> Hash of the key - here we used a random UUID as the key - still a bad
> distribution
>
>
> [image: image.png]
>
>
>
> When I start, let's say 10 producers, we can get 5 messages going to one
> partition while others have none - i understand that this is because the
> round robin is internal to the producer.
> [image: image.png]
>
> (The picture above is what I believe is happening)
>
>
> it would surprise me that this hasn't come up before, that's why I'm
> pretty sure I'm missing something here...
> We're currently analyzing some solutions, one of them is building our own
> partitioner with shared memory (yes, that's how far we got on this),
> although I believe a better solution would be to have this on Kafka broker
> side and not depend on custom code.
>
> [image: image.png]
> Above is the draft of one of our current ideas of a possible design. Based
> on the shared memory we could decide the partition and send the messages
> directly there (the number of producers, partitions and consumers here are
> simplified).
> This if we don't find a solution within Kafka to mitigate this
> distribution issue - even though the thought design imposes limits and add
> layers we didn't had in our initial design.
>
> My 

Re: Broker side partition round robin

[Liam Clarke-Hutchinson]

Hi, I can't see your images, they're not making it through the mailing
list, sorry.

On Tue, 2 Jun. 2020, 11:09 pm Vinicius Scheidegger, <
vinicius.scheideg...@gmail.com> wrote:

> Hi Liam,
>
> First of all, thanks for checking my doubt.
> I understand that the reason I notice this behavior is because our case
> differs from yours in one point: I'm not writing terabytes in a single day.
> I'm writing MB, but distribution matters in these MBs, this
> because processing in the consumer side is CPU intensive (complex machine
> learning algo), so a real equally distributed load is a requirement,
> otherwise messages start queuing (which is what I don't want).
>
> Our system is also in production and this bad distribution is generating
> queuing in some consumers while others are idle.
> I tried both approaches:
> Round Robbin - actually gives me a distributed load in a big time window,
> but as I grow the number of producers I get several messages in a single
> partition while others don't get anything.
> Hash of the key - here we used a random UUID as the key - still a bad
> distribution
>
>
> [image: image.png]
>
>
>
> When I start, let's say 10 producers, we can get 5 messages going to one
> partition while others have none - i understand that this is because the
> round robin is internal to the producer.
> [image: image.png]
>
> (The picture above is what I believe is happening)
>
>
> it would surprise me that this hasn't come up before, that's why I'm
> pretty sure I'm missing something here...
> We're currently analyzing some solutions, one of them is building our own
> partitioner with shared memory (yes, that's how far we got on this),
> although I believe a better solution would be to have this on Kafka broker
> side and not depend on custom code.
>
> [image: image.png]
> Above is the draft of one of our current ideas of a possible design. Based
> on the shared memory we could decide the partition and send the messages
> directly there (the number of producers, partitions and consumers here are
> simplified).
> This if we don't find a solution within Kafka to mitigate this
> distribution issue - even though the thought design imposes limits and add
> layers we didn't had in our initial design.
>
> My question is, do we really need to develop this??
> Is equal distribution in a scenario with multiple producers something that
> can be achieved in Kafka without custom development?
> Having never checked out the broker code when receiving a message - is
> partition definition even possible on the broker side?
> If this really does not exist, would a feature like that benefit other
> people, is it worth checking out instead of the above solution?
> Should I move this message to the dev forum? - no one gave me much
> attention there too (but maybe my messages are too big/boring - hahaha)
>
> Thanks
>
>
> On Tue, Jun 2, 2020 at 10:47 AM Liam Clarke-Hutchinson <
> liam.cla...@adscale.co.nz> wrote:
>
>> Hi Vinicius,
>>
>> As you note, the cluster doesn't load balance producers, it relies on them
>> using a partition strategy to do so.
>>
>> In production, I've never had actual broker load skew develop from
>> multiple
>> independent producers using round robining - and we're talking say 20 - 50
>> producers (depending on scaling) writing terabytes over a day.
>>
>> And load skew / hot brokers is something I monitor closely.
>>
>> The only time I've seen load skew is when a key based partition strategy
>> was used, and keys weren't evenly distributed.
>>
>> So in other words, in theory there's no guarantee, but in my experience,
>> round robining multiple producers works fine.
>>
>> Cheers,
>>
>> Liam Clarke
>>
>> On Mon, 1 Jun. 2020, 11:55 pm Vinicius Scheidegger, <
>> vinicius.scheideg...@gmail.com> wrote:
>>
>> > Hey guys, I need some help here...
>> >
>> > Is this a flaw in the design (maybe a discussion point for a KIP?), is
>> > Kafka not supposed to perform equal load balancing with multiple
>> producers
>> > or am I missing something (which is what I believe is happening)?
>> >
>> > On Wed, May 27, 2020 at 2:40 PM Vinicius Scheidegger <
>> > vinicius.scheideg...@gmail.com> wrote:
>> >
>> >> Does anyone know whether we could really have an "out of the box"
>> >> solution to do round robin over the partitions when we have multiple
>> >> producers?
>> >> By that I mean, a round robin on the broker side (or maybe some way to
>> >> synchronize all producers).
>> >>
>> >> Thank you,
>> >>
>> >> On Tue, May 26, 2020 at 1:41 PM Vinicius Scheidegger <
>> >> vinicius.scheideg...@gmail.com> wrote:
>> >>
>> >>> Yes, I checked it. The issue is that RoundRobbinPartitioner is bound
>> to
>> >>> the producer. In a scenario with multiple producers it doesn't
>> guarantee
>> >>> equal distribution - from what I understood and from my tests, the
>> >>> following situation happens with it:
>> >>>
>> >>> [image: image.png]
>> >>>
>> >>> Of course, the first partition is not always 1 and each producer may
>> >>> start in a 

Re: Broker side partition round robin

[Vinicius Scheidegger]

Hi Liam,

First of all, thanks for checking my doubt.
I understand that the reason I notice this behavior is because our case
differs from yours in one point: I'm not writing terabytes in a single day.
I'm writing MB, but distribution matters in these MBs, this
because processing in the consumer side is CPU intensive (complex machine
learning algo), so a real equally distributed load is a requirement,
otherwise messages start queuing (which is what I don't want).

Our system is also in production and this bad distribution is generating
queuing in some consumers while others are idle.
I tried both approaches:
Round Robbin - actually gives me a distributed load in a big time window,
but as I grow the number of producers I get several messages in a single
partition while others don't get anything.
Hash of the key - here we used a random UUID as the key - still a bad
distribution


[image: image.png]



When I start, let's say 10 producers, we can get 5 messages going to one
partition while others have none - i understand that this is because the
round robin is internal to the producer.
[image: image.png]

(The picture above is what I believe is happening)


it would surprise me that this hasn't come up before, that's why I'm pretty
sure I'm missing something here...
We're currently analyzing some solutions, one of them is building our own
partitioner with shared memory (yes, that's how far we got on this),
although I believe a better solution would be to have this on Kafka broker
side and not depend on custom code.

[image: image.png]
Above is the draft of one of our current ideas of a possible design. Based
on the shared memory we could decide the partition and send the messages
directly there (the number of producers, partitions and consumers here are
simplified).
This if we don't find a solution within Kafka to mitigate this distribution
issue - even though the thought design imposes limits and add layers we
didn't had in our initial design.

My question is, do we really need to develop this??
Is equal distribution in a scenario with multiple producers something that
can be achieved in Kafka without custom development?
Having never checked out the broker code when receiving a message - is
partition definition even possible on the broker side?
If this really does not exist, would a feature like that benefit other
people, is it worth checking out instead of the above solution?
Should I move this message to the dev forum? - no one gave me much
attention there too (but maybe my messages are too big/boring - hahaha)

Thanks


On Tue, Jun 2, 2020 at 10:47 AM Liam Clarke-Hutchinson <
liam.cla...@adscale.co.nz> wrote:

> Hi Vinicius,
>
> As you note, the cluster doesn't load balance producers, it relies on them
> using a partition strategy to do so.
>
> In production, I've never had actual broker load skew develop from multiple
> independent producers using round robining - and we're talking say 20 - 50
> producers (depending on scaling) writing terabytes over a day.
>
> And load skew / hot brokers is something I monitor closely.
>
> The only time I've seen load skew is when a key based partition strategy
> was used, and keys weren't evenly distributed.
>
> So in other words, in theory there's no guarantee, but in my experience,
> round robining multiple producers works fine.
>
> Cheers,
>
> Liam Clarke
>
> On Mon, 1 Jun. 2020, 11:55 pm Vinicius Scheidegger, <
> vinicius.scheideg...@gmail.com> wrote:
>
> > Hey guys, I need some help here...
> >
> > Is this a flaw in the design (maybe a discussion point for a KIP?), is
> > Kafka not supposed to perform equal load balancing with multiple
> producers
> > or am I missing something (which is what I believe is happening)?
> >
> > On Wed, May 27, 2020 at 2:40 PM Vinicius Scheidegger <
> > vinicius.scheideg...@gmail.com> wrote:
> >
> >> Does anyone know whether we could really have an "out of the box"
> >> solution to do round robin over the partitions when we have multiple
> >> producers?
> >> By that I mean, a round robin on the broker side (or maybe some way to
> >> synchronize all producers).
> >>
> >> Thank you,
> >>
> >> On Tue, May 26, 2020 at 1:41 PM Vinicius Scheidegger <
> >> vinicius.scheideg...@gmail.com> wrote:
> >>
> >>> Yes, I checked it. The issue is that RoundRobbinPartitioner is bound to
> >>> the producer. In a scenario with multiple producers it doesn't
> guarantee
> >>> equal distribution - from what I understood and from my tests, the
> >>> following situation happens with it:
> >>>
> >>> [image: image.png]
> >>>
> >>> Of course, the first partition is not always 1 and each producer may
> >>> start in a different point in time, anyway my point is that it does not
> >>> guarantee equal distribution.
> >>>
> >>> The other option pointed out is to select the partition myself - either
> >>> a shared memory on the producers (assuming that this is possible - I
> mean I
> >>> would need to guarantee that producers CAN share a synchronized
> memory) or
> 

Re: Broker side partition round robin

[Liam Clarke-Hutchinson]

Hi Vinicius,

As you note, the cluster doesn't load balance producers, it relies on them
using a partition strategy to do so.

In production, I've never had actual broker load skew develop from multiple
independent producers using round robining - and we're talking say 20 - 50
producers (depending on scaling) writing terabytes over a day.

And load skew / hot brokers is something I monitor closely.

The only time I've seen load skew is when a key based partition strategy
was used, and keys weren't evenly distributed.

So in other words, in theory there's no guarantee, but in my experience,
round robining multiple producers works fine.

Cheers,

Liam Clarke

On Mon, 1 Jun. 2020, 11:55 pm Vinicius Scheidegger, <
vinicius.scheideg...@gmail.com> wrote:

> Hey guys, I need some help here...
>
> Is this a flaw in the design (maybe a discussion point for a KIP?), is
> Kafka not supposed to perform equal load balancing with multiple producers
> or am I missing something (which is what I believe is happening)?
>
> On Wed, May 27, 2020 at 2:40 PM Vinicius Scheidegger <
> vinicius.scheideg...@gmail.com> wrote:
>
>> Does anyone know whether we could really have an "out of the box"
>> solution to do round robin over the partitions when we have multiple
>> producers?
>> By that I mean, a round robin on the broker side (or maybe some way to
>> synchronize all producers).
>>
>> Thank you,
>>
>> On Tue, May 26, 2020 at 1:41 PM Vinicius Scheidegger <
>> vinicius.scheideg...@gmail.com> wrote:
>>
>>> Yes, I checked it. The issue is that RoundRobbinPartitioner is bound to
>>> the producer. In a scenario with multiple producers it doesn't guarantee
>>> equal distribution - from what I understood and from my tests, the
>>> following situation happens with it:
>>>
>>> [image: image.png]
>>>
>>> Of course, the first partition is not always 1 and each producer may
>>> start in a different point in time, anyway my point is that it does not
>>> guarantee equal distribution.
>>>
>>> The other option pointed out is to select the partition myself - either
>>> a shared memory on the producers (assuming that this is possible - I mean I
>>> would need to guarantee that producers CAN share a synchronized memory) or
>>> include an intermediate topic with a single partition and a
>>> dispatcher/producer using RoundRobinPartitioner (but this would include a
>>> single point of failure).
>>>
>>> [image: image.png]
>>> [image: image.png]
>>>
>>> None of these seem to be ideal as a Broker side round robin solution
>>> would.
>>> Am I missing something? Any other ideas?
>>>
>>> Thanks
>>>
>>> On Tue, May 26, 2020 at 11:34 AM M. Manna  wrote:
>>>
 Hey Vinicius,


 On Tue, 26 May 2020 at 10:27, Vinicius Scheidegger <
 vinicius.scheideg...@gmail.com> wrote:

 > In a scenario with multiple independent producers (imagine ephemeral
 > dockers, that do not know the state of each other), what should be the
 > approach for the messages being sent to be equally distributed over a
 topic
 > partition?
 >
 > From what I understood the partition election is always on the
 Producer. Is
 > this understanding correct?
 >
 > If that's the case, how should one achieve an equally distributed load
 > balancing (round robin) over the partitions in a scenario with
 multiple
 > producers?
 >
 > Thank you,
 >
 > Vinicius Scheidegger


  Have you checked RoundRobinPartitioner ? Also, you can always specify
 which partition you are writing to, so you can control the partitioning
 in
 your way.

 Regards,

 Regards,

 >
 >

>>>

Re: Broker side partition round robin

[Vinicius Scheidegger]

Hey guys, I need some help here...

Is this a flaw in the design (maybe a discussion point for a KIP?), is
Kafka not supposed to perform equal load balancing with multiple producers
or am I missing something (which is what I believe is happening)?

On Wed, May 27, 2020 at 2:40 PM Vinicius Scheidegger <
vinicius.scheideg...@gmail.com> wrote:

> Does anyone know whether we could really have an "out of the box" solution
> to do round robin over the partitions when we have multiple producers?
> By that I mean, a round robin on the broker side (or maybe some way to
> synchronize all producers).
>
> Thank you,
>
> On Tue, May 26, 2020 at 1:41 PM Vinicius Scheidegger <
> vinicius.scheideg...@gmail.com> wrote:
>
>> Yes, I checked it. The issue is that RoundRobbinPartitioner is bound to
>> the producer. In a scenario with multiple producers it doesn't guarantee
>> equal distribution - from what I understood and from my tests, the
>> following situation happens with it:
>>
>> [image: image.png]
>>
>> Of course, the first partition is not always 1 and each producer may
>> start in a different point in time, anyway my point is that it does not
>> guarantee equal distribution.
>>
>> The other option pointed out is to select the partition myself - either a
>> shared memory on the producers (assuming that this is possible - I mean I
>> would need to guarantee that producers CAN share a synchronized memory) or
>> include an intermediate topic with a single partition and a
>> dispatcher/producer using RoundRobinPartitioner (but this would include a
>> single point of failure).
>>
>> [image: image.png]
>> [image: image.png]
>>
>> None of these seem to be ideal as a Broker side round robin solution
>> would.
>> Am I missing something? Any other ideas?
>>
>> Thanks
>>
>> On Tue, May 26, 2020 at 11:34 AM M. Manna  wrote:
>>
>>> Hey Vinicius,
>>>
>>>
>>> On Tue, 26 May 2020 at 10:27, Vinicius Scheidegger <
>>> vinicius.scheideg...@gmail.com> wrote:
>>>
>>> > In a scenario with multiple independent producers (imagine ephemeral
>>> > dockers, that do not know the state of each other), what should be the
>>> > approach for the messages being sent to be equally distributed over a
>>> topic
>>> > partition?
>>> >
>>> > From what I understood the partition election is always on the
>>> Producer. Is
>>> > this understanding correct?
>>> >
>>> > If that's the case, how should one achieve an equally distributed load
>>> > balancing (round robin) over the partitions in a scenario with multiple
>>> > producers?
>>> >
>>> > Thank you,
>>> >
>>> > Vinicius Scheidegger
>>>
>>>
>>>  Have you checked RoundRobinPartitioner ? Also, you can always specify
>>> which partition you are writing to, so you can control the partitioning
>>> in
>>> your way.
>>>
>>> Regards,
>>>
>>> Regards,
>>>
>>> >
>>> >
>>>
>>

Re: Broker side partition round robin

[Vinicius Scheidegger]

Does anyone know whether we could really have an "out of the box" solution
to do round robin over the partitions when we have multiple producers?
By that I mean, a round robin on the broker side (or maybe some way to
synchronize all producers).

Thank you,

On Tue, May 26, 2020 at 1:41 PM Vinicius Scheidegger <
vinicius.scheideg...@gmail.com> wrote:

> Yes, I checked it. The issue is that RoundRobbinPartitioner is bound to
> the producer. In a scenario with multiple producers it doesn't guarantee
> equal distribution - from what I understood and from my tests, the
> following situation happens with it:
>
> [image: image.png]
>
> Of course, the first partition is not always 1 and each producer may start
> in a different point in time, anyway my point is that it does not guarantee
> equal distribution.
>
> The other option pointed out is to select the partition myself - either a
> shared memory on the producers (assuming that this is possible - I mean I
> would need to guarantee that producers CAN share a synchronized memory) or
> include an intermediate topic with a single partition and a
> dispatcher/producer using RoundRobinPartitioner (but this would include a
> single point of failure).
>
> [image: image.png]
> [image: image.png]
>
> None of these seem to be ideal as a Broker side round robin solution would.
> Am I missing something? Any other ideas?
>
> Thanks
>
> On Tue, May 26, 2020 at 11:34 AM M. Manna  wrote:
>
>> Hey Vinicius,
>>
>>
>> On Tue, 26 May 2020 at 10:27, Vinicius Scheidegger <
>> vinicius.scheideg...@gmail.com> wrote:
>>
>> > In a scenario with multiple independent producers (imagine ephemeral
>> > dockers, that do not know the state of each other), what should be the
>> > approach for the messages being sent to be equally distributed over a
>> topic
>> > partition?
>> >
>> > From what I understood the partition election is always on the
>> Producer. Is
>> > this understanding correct?
>> >
>> > If that's the case, how should one achieve an equally distributed load
>> > balancing (round robin) over the partitions in a scenario with multiple
>> > producers?
>> >
>> > Thank you,
>> >
>> > Vinicius Scheidegger
>>
>>
>>  Have you checked RoundRobinPartitioner ? Also, you can always specify
>> which partition you are writing to, so you can control the partitioning in
>> your way.
>>
>> Regards,
>>
>> Regards,
>>
>> >
>> >
>>
>

Re: Broker side partition round robin

[Vinicius Scheidegger]

Yes, I checked it. The issue is that RoundRobbinPartitioner is bound to the
producer. In a scenario with multiple producers it doesn't guarantee equal
distribution - from what I understood and from my tests, the following
situation happens with it:

[image: image.png]

Of course, the first partition is not always 1 and each producer may start
in a different point in time, anyway my point is that it does not guarantee
equal distribution.

The other option pointed out is to select the partition myself - either a
shared memory on the producers (assuming that this is possible - I mean I
would need to guarantee that producers CAN share a synchronized memory) or
include an intermediate topic with a single partition and a
dispatcher/producer using RoundRobinPartitioner (but this would include a
single point of failure).

[image: image.png]
[image: image.png]

None of these seem to be ideal as a Broker side round robin solution would.
Am I missing something? Any other ideas?

Thanks

On Tue, May 26, 2020 at 11:34 AM M. Manna  wrote:

> Hey Vinicius,
>
>
> On Tue, 26 May 2020 at 10:27, Vinicius Scheidegger <
> vinicius.scheideg...@gmail.com> wrote:
>
> > In a scenario with multiple independent producers (imagine ephemeral
> > dockers, that do not know the state of each other), what should be the
> > approach for the messages being sent to be equally distributed over a
> topic
> > partition?
> >
> > From what I understood the partition election is always on the Producer.
> Is
> > this understanding correct?
> >
> > If that's the case, how should one achieve an equally distributed load
> > balancing (round robin) over the partitions in a scenario with multiple
> > producers?
> >
> > Thank you,
> >
> > Vinicius Scheidegger
>
>
>  Have you checked RoundRobinPartitioner ? Also, you can always specify
> which partition you are writing to, so you can control the partitioning in
> your way.
>
> Regards,
>
> Regards,
>
> >
> >
>

Re: Broker side partition round robin

[M. Manna]

Hey Vinicius,


On Tue, 26 May 2020 at 10:27, Vinicius Scheidegger <
vinicius.scheideg...@gmail.com> wrote:

> In a scenario with multiple independent producers (imagine ephemeral
> dockers, that do not know the state of each other), what should be the
> approach for the messages being sent to be equally distributed over a topic
> partition?
>
> From what I understood the partition election is always on the Producer. Is
> this understanding correct?
>
> If that's the case, how should one achieve an equally distributed load
> balancing (round robin) over the partitions in a scenario with multiple
> producers?
>
> Thank you,
>
> Vinicius Scheidegger


 Have you checked RoundRobinPartitioner ? Also, you can always specify
which partition you are writing to, so you can control the partitioning in
your way.

Regards,

Regards,

>
>