Hi Gavin, The problem is that the Kinesis producer currently does not propagate backpressure properly. Records are added to the internally used KPL client’s queue, without any queue size limit.
This is considered a bug, and already has a pull request for it [1], which we should probably push towards being merged soon. What the pull request essentially does, is adding an upper bound to the number pending records in the KPL producer queue. Once the upper bound is hit, input to the Kinesis producer sink is blocked, and therefore propagating backpressure further upstream. Cheers, Gordon [1] https://github.com/apache/flink/pull/6021 On 20 June 2018 at 6:00:30 PM, Liu, Gavin (CAI - Atlanta) (gavin....@coxautoinc.com) wrote: Hi guys, I am new to flink framework. And we are building an application that takes kinesis stream for both flink source and sink. The flink version we are using is 1.4.2, which is also the version for the flink-connector-kinesis. We built the flink-connector-kinesis jar explicitly with KPL version 0.12.6 due to the existing problems with default 0.12.5. I get a rough idea how the backpressure works with flink through reading http://mail-archives.apache.org/mod_mbox/flink-user/201801.mbox/%3cf8dd76c0-9de0-412a-8c24-b72af0d42...@data-artisans.com%3E From my experiment with flink and flink-connector-kinesis, the back pressure only happens within flink processing operations, i.e., not in the flink producer to kinesis. More specifically, when the throughput from KPL exceeds the kinesis throughput limitations, flink does not slow down at all, i.e., it does not add pressure on the processing chain up to the flink consumer. Correct me if I misunderstood this. It looks like the flink producer (in the flink-connector-kinesis) is a standalone component, once a record is collected and sent to the producer, flink core finishes all the processing and does not care the fate of the record any more, it is the responsibility of the connector to continue the job. I am expecting back pressure to happen from the source kinesis stream to the sink kinesis stream, whenever the sink kinesis stream could not handle the volume, it adds back pressure. Could someone illustrate a bit more why flink connector is designed in such a way. Also correct me if I stated anything wrong. Gavin Liu
