Bowen Li created FLINK-7508:
-------------------------------
Summary: switch FlinkKinesisProducer to use KPL's ThreadingMode to
ThreadedPool mode rather than Per_Request mode
Key: FLINK-7508
URL: https://issues.apache.org/jira/browse/FLINK-7508
Project: Flink
Issue Type: Improvement
Components: Kinesis Connector
Affects Versions: 1.3.0
Reporter: Bowen Li
Assignee: Bowen Li
Fix For: 1.4.0, 1.3.3
KinesisProducerLibrary (KPL) 0.10.x had been using a One-New-Thread-Per-Request
model for all requests sent to AWS Kinesis, which is very expensive.
0.12.4 introduced a new [ThreadingMode -
Pooled|https://github.com/awslabs/amazon-kinesis-producer/blob/master/java/amazon-kinesis-producer/src/main/java/com/amazonaws/services/kinesis/producer/KinesisProducerConfiguration.java#L225],
which will use a thread pool. This hugely improves KPL's performance and
reduces consumed resources. By default, KPL still uses per-request mode. We
should explicitly switch FlinkKinesisProducer's KPL threading mode to 'Pooled'.
This work depends on FLINK-7366 which upgrades KPL from 0.10 to 0.12.5.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
