[
https://issues.apache.org/jira/browse/FLINK-7622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tzu-Li (Gordon) Tai updated FLINK-7622:
---------------------------------------
Description:
This issue was brought to discussion by [~sthm] offline.
Currently, records are added to the Kinesis KPL producer client without
checking the number of outstanding records within the local KPL queue. This
manner is basically neglecting backpressure when producing to Kinesis through
KPL, and can therefore exhaust system resources.
We should respect {{producer.getOutstandingRecordsCount()}} as a measure of
backpressure, and propagate backpressure upstream by blocking further sink
invocations when some threshold of outstanding record count is exceeded. The
recommended threshold [1] seems to be 10,000.
[1]
https://aws.amazon.com/blogs/big-data/implementing-efficient-and-reliable-producers-with-the-amazon-kinesis-producer-library/
was:
This issue was brought to discussion by [~sthm] offline.
Currently, records are added to the Kinesis KPL producer client without
checking the number of outstanding records within the local KPL queue. This
manner is basically neglecting backpressure when producing to Kinesis through
KPL, and can therefore exhaust system resources.
We should respect {{producer.getOutstandingRecordsCount()}} as a measure of
backpressure, and propagate backpressure upstream by blocking further sink
invocations when some threshold of outstanding record count is exceeded. The
recommended threshold [1] seems to be 10,000.
> Respect local KPL queue size in FlinkKinesisProducer when adding records to
> KPL client
> --------------------------------------------------------------------------------------
>
> Key: FLINK-7622
> URL: https://issues.apache.org/jira/browse/FLINK-7622
> Project: Flink
> Issue Type: Improvement
> Components: Kinesis Connector
> Reporter: Tzu-Li (Gordon) Tai
>
> This issue was brought to discussion by [~sthm] offline.
> Currently, records are added to the Kinesis KPL producer client without
> checking the number of outstanding records within the local KPL queue. This
> manner is basically neglecting backpressure when producing to Kinesis through
> KPL, and can therefore exhaust system resources.
> We should respect {{producer.getOutstandingRecordsCount()}} as a measure of
> backpressure, and propagate backpressure upstream by blocking further sink
> invocations when some threshold of outstanding record count is exceeded. The
> recommended threshold [1] seems to be 10,000.
> [1]
> https://aws.amazon.com/blogs/big-data/implementing-efficient-and-reliable-producers-with-the-amazon-kinesis-producer-library/
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
