Hi Ajay, Flink handles "backpressure" in a graceful way so that it doesn't get affected when your processing pipeline is occasionally slowed down. I think the following articles will help [1,2].
In your specific case: the "KeyBy" operation will re-hash data so they can be reshuffled from all input consumers to all your process operators (in this case the MyKeyedProcessFunction). If one of the process operator is backpressured, it will back track all the way to the source. So, my understanding is that: since there's the reshuffling, if one of the process function is backpressured, it will potentially affect all the source operators. Thanks, Rong [1] https://www.ververica.com/blog/how-flink-handles-backpressure [2] https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/back_pressure.html On Wed, Feb 13, 2019 at 8:50 AM Aggarwal, Ajay <ajay.aggar...@netapp.com> wrote: > I was wondering what is the impact if one of the stream operator function > occasionally takes too long to process the event. Given the following > simple flink job > > > > inputStream > > .KeyBy (“tenantId”) > > .process ( new MyKeyedProcessFunction()) > > > > , if occasionally MyKeyedProcessFunction takes too long (say ~5-10 > minutes) to process an incoming element, what is the impact on overall > pipeline? Is the impact limited to > > 1. Specific key for which MyKeyedProcessFunction is currently taking > too long to process an element, or > 2. Specific Taskslot, where MyKeyedProcessFunction is currently taking > too long to process an element, i.e. impacting multiple keys, or > 3. Entire inputstream ? > > > > Also what is the built in resiliency in these cases? Is there a concept of > timeout for each operator function? > > > > Ajay >
