[
https://issues.apache.org/jira/browse/FLINK-7367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117350#comment-16117350
]
ASF GitHub Bot commented on FLINK-7367:
---------------------------------------
Github user bowenli86 commented on the issue:
https://github.com/apache/flink/pull/4473
Hi @tzulitai ,
Let's reach a consensus before I do any more works.
1) I didn't add all KPL's configs in this PR. I only added some
configs that I think might be useful to Flink users. The full list of KPL
configs is
[here](https://github.com/awslabs/amazon-kinesis-producer/blob/master/java/amazon-kinesis-producer-sample/default_config.properties).
I don't think exposing all KPL configs is a good idea. What do you think?
2) I absolutely agree that there'd better be a consistent way to keep
configs up-tp-date. But there probably isn't as long as Flink wraps around KPL,
because right now we have to always set KPL configs in
`FlinkKinesisProducer#open()` with `setXxxx()`
3) If Flink uses
[`KinesisProducerConfiguration#fromProperties(Properties
props)`](https://github.com/awslabs/amazon-kinesis-producer/blob/master/java/amazon-kinesis-producer/src/main/java/com/amazonaws/services/kinesis/producer/KinesisProducerConfiguration.java)
to init `KinesisProducerConfiguration`, users can always pass in the configs
as k-v string pairs. But, it requires changing all config values in
`ProducerConfigConstants`, e.g. from `aws.producer.aggregationMaxCount` to
`AggregationMaxCount`, because KPL library itself doesn't provide a util class
with key names as string. How much would the impact be to rename those config
keys?
Bowen
> Parameterize more configs for FlinkKinesisProducer (RecordMaxBufferedTime,
> MaxConnections, RequestTimeout, etc)
> ---------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-7367
> URL: https://issues.apache.org/jira/browse/FLINK-7367
> Project: Flink
> Issue Type: Improvement
> Components: Kinesis Connector
> Affects Versions: 1.3.0
> Reporter: Bowen Li
> Assignee: Bowen Li
> Fix For: 1.3.3
>
>
> Right now, FlinkKinesisProducer only expose two configs for the underlying
> KinesisProducer:
> - AGGREGATION_MAX_COUNT
> - COLLECTION_MAX_COUNT
> Well, according to [AWS
> doc|http://docs.aws.amazon.com/streams/latest/dev/kinesis-kpl-config.html]
> and [their sample on
> github|https://github.com/awslabs/amazon-kinesis-producer/blob/master/java/amazon-kinesis-producer-sample/default_config.properties],
> developers can set more to make the max use of KinesisProducer, and make it
> fault-tolerant (e.g. by increasing timeout).
> I select a few more configs that we need when using Flink with Kinesis:
> - MAX_CONNECTIONS
> - RATE_LIMIT
> - RECORD_MAX_BUFFERED_TIME
> - RECORD_TIME_TO_LIVE
> - REQUEST_TIMEOUT
> We need to parameterize FlinkKinesisProducer to pass in the above params, in
> order to cater to our need
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
