Re: synchronizing streams of different kafka topics

[Cody Koeninger]

Are you using the direct stream? Each batch should contain all of the
unprocessed messages for each topic, unless you're doing some kind of rate
limiting.

On Tue, Nov 17, 2015 at 3:07 AM, Antony Mayi 
wrote:

> Hi,
>
> I have two streams coming from two different kafka topics. the two topics
> contain time related events but are quite asymmetric in volume. I would
> obviously need to process them in sync to get the time related events
> together but with same processing rate if the heavier stream starts
> backlogging the events from the tinier stream would be coming ahead of the
> relevant events that are still in the backlog of the heavy stream.
>
> Is there any way to get the smaller stream processed with slower rate so
> that the relevant events come together with the heavy stream?
>
> Thanks,
> Antony.
>

synchronizing streams of different kafka topics

[Antony Mayi]

Hi,
I have two streams coming from two different kafka topics. the two topics 
contain time related events but are quite asymmetric in volume. I would 
obviously need to process them in sync to get the time related events together 
but with same processing rate if the heavier stream starts backlogging the 
events from the tinier stream would be coming ahead of the relevant events that 
are still in the backlog of the heavy stream.
Is there any way to get the smaller stream processed with slower rate so that 
the relevant events come together with the heavy stream?
Thanks,Antony.