ASF GitHub Bot commented on FLINK-7366:

Github user aljoscha commented on the issue:

    @tzulitai Are we not using Java Serialisation starting from 1.2.x?

> Upgrade kinesis producer library, kinesis client library, and AWS SDK in 
> flink-connector-kinesis
> ------------------------------------------------------------------------------------------------
>                 Key: FLINK-7366
>                 URL: https://issues.apache.org/jira/browse/FLINK-7366
>             Project: Flink
>          Issue Type: Bug
>          Components: Kinesis Connector
>    Affects Versions: 1.3.2
>            Reporter: Bowen Li
>            Assignee: Bowen Li
>             Fix For: 1.4.0, 1.3.3
> We need to upgrade KPL and KCL to pick up the enhanced performance and 
> stability for Flink to work better  with Kinesis. Upgrading KPL is specially 
> necessary, because the KPL version Flink uses is old, and doesn't have good 
> retry and error handling logic.
> *KPL:*
> flink-connector-kinesis currently uses kinesis-producer-library 0.10.2, which 
> is released in Nov 2015 by AWS. It's old. It's the fourth release, and thus 
> problematic. It doesn't even have good retry logic, therefore Flink fails 
> really frequently (about every 10 mins as we observed) when Flink writes too 
> fast to Kinesis and receives RateLimitExceededException, 
> Quotes from https://github.com/awslabs/amazon-kinesis-producer/issues/56, 
> "*With the newer version of the KPL it uses the AWS C++ SDK which should 
> offer additional retries.*" on Oct 2016. 0.12.5, the version we are upgrading 
> to, is released in May 2017 and should have the enhanced retry logic.
> *KCL:*
> Upgrade KCL from 1.6.2 to 1.8.1
> from 1.10.71 to 1.11.171

This message was sent by Atlassian JIRA

Reply via email to