Can you explain how back pressure affect the source in flink? I read the
great article
https://data-artisans.com/blog/how-flink-handles-backpressure and got the
idea but I would like to know more details. Let's consider

org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext


interface and its

void collect(T element);

method.


Is back pressure mechanism going to to block the calling collect method
thread for some time?

How does it compare what has been written in the mentioned article? I don't
quite understand how  '*The output side never puts too much data on the
wire by a simple watermark mechanism*' is supposed to work.

   - *Remote exchange*: If task 1 and task 2 run on different worker nodes,
   the buffer can be recycled as soon as it is on the wire (TCP channel). On
   the receiving side, the data is copied from the wire to a buffer from the
   input buffer pool. If no buffer is available, reading from the TCP
   connection is interrupted. The output side never puts too much data on the
   wire by a simple watermark mechanism. If enough data is in-flight, we wait
   before we copy more data to the wire until it is below a threshold. This
   guarantees that there is never too much data in-flight. If new data is not
   consumed on the receiving side (because there is no buffer available), this
   slows down the sender.

Thanks,
Pawel

Reply via email to