[ https://issues.apache.org/jira/browse/FLINK-7367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16113914#comment-16113914 ]
ASF GitHub Bot commented on FLINK-7367: --------------------------------------- GitHub user bowenli86 opened a pull request: https://github.com/apache/flink/pull/4473 [FLINK-7367][kinesis connector] Parameterize more configs for FlinkKinesisProducer (RecordMaxBufferedTime, MaxConnections, RequestTimeout, etc) ## What is the purpose of the change Right now, FlinkKinesisProducer only expose two configs for the underlying KinesisProducer: - AGGREGATION_MAX_COUNT - COLLECTION_MAX_COUNT Well, according to AWS doc and their sample on github, developers can set more to make the max use of KinesisProducer, and make it fault-tolerant (e.g. by increasing timeout). I select a few more configs that we need when using Flink with Kinesis: - MAX_CONNECTIONS - RATE_LIMIT - RECORD_MAX_BUFFERED_TIME - RECORD_TIME_TO_LIVE - REQUEST_TIMEOUT We need to parameterize FlinkKinesisProducer to pass in the above params, in order to cater to our need ## Brief change log - *Added more config values into `ProducerConfigConstants`* - *Made FlinkKinesisProducer pick up more configs* - *Added an example in doc* ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) ## Documentation - Does this pull request introduce a new feature? (no) You can merge this pull request into a Git repository by running: $ git pull https://github.com/bowenli86/flink FLINK-7363 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/4473.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4473 ---- commit ac0b4d22fde763dde62b26ed9a022d537bb29e58 Author: Bowen Li <bowenl...@gmail.com> Date: 2017-08-04T03:59:02Z FLINK-7367 Parameterize FlinkKinesisProducer on RecordMaxBufferedTime, MaxConnections, RequestTimeout, and more ---- > Parameterize more configs for FlinkKinesisProducer (RecordMaxBufferedTime, > MaxConnections, RequestTimeout, etc) > --------------------------------------------------------------------------------------------------------------- > > Key: FLINK-7367 > URL: https://issues.apache.org/jira/browse/FLINK-7367 > Project: Flink > Issue Type: Improvement > Components: Kinesis Connector > Affects Versions: 1.3.0 > Reporter: Bowen Li > Assignee: Bowen Li > Fix For: 1.3.3 > > > Right now, FlinkKinesisProducer only expose two configs for the underlying > KinesisProducer: > - AGGREGATION_MAX_COUNT > - COLLECTION_MAX_COUNT > Well, according to [AWS > doc|http://docs.aws.amazon.com/streams/latest/dev/kinesis-kpl-config.html] > and [their sample on > github|https://github.com/awslabs/amazon-kinesis-producer/blob/master/java/amazon-kinesis-producer-sample/default_config.properties], > developers can set more to make the max use of KinesisProducer, and make it > fault-tolerant (e.g. by increasing timeout). > I select a few more configs that we need when using Flink with Kinesis: > - MAX_CONNECTIONS > - RATE_LIMIT > - RECORD_MAX_BUFFERED_TIME > - RECORD_TIME_TO_LIVE > - REQUEST_TIMEOUT > We need to parameterize FlinkKinesisProducer to pass in the above params, in > order to cater to our need -- This message was sent by Atlassian JIRA (v6.4.14#64029)