Yes, my job does do a keyBy. It never occurred to me that keyBy would
distributed data from different partitions to different tasks, but, now
that you mention it, it actually makes perfect sense. Thanks you for the
help.
On Thu, Oct 4, 2018 at 5:11 PM Elias Levy
wrote:
> Does your job perform
Hey Andrew,
We face the same problem in our application where some of the
kafka partitions are empty. In this case what we do is use the rebalance()
method on the source streams.
Ex: DataStream srcStream= Env.addSource(new
FlinkKafkaConsumer09<>(topic,Ser,props));
SrcStream.rebalance();
Does your job perform a keyBy or broadcast that would result in data from
different partitions being distributed among tasks? If so, then that would
be the cause.
On Thu, Oct 4, 2018 at 12:58 PM Andrew Kowpak
wrote:
> Hi all,
>
> I apologize if this has been discussed to death in the past,
Hi all,
I apologize if this has been discussed to death in the past, but, I'm
finding myself very confused, and google is not proving helpful.
Based on the documentation, I understand that if there are idle partitions
in a kafka stream, watermarks will not advance for the entire application.
I
