[
https://issues.apache.org/jira/browse/FLINK-12847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16866585#comment-16866585
]
Dyana Rose commented on FLINK-12847:
------------------------------------
Working branch for reference:
compare:
[https://github.com/apache/flink/compare/master...SaleCycle:FLINK-12847]
branch: [https://github.com/SaleCycle/flink/tree/FLINK-12847]
Migration docs for reference:
[https://docs.aws.amazon.com/sdk-for-java/v2/migration-guide/whats-different.html]
[https://docs.aws.amazon.com/streams/latest/dev/kcl-migration.html]
[https://docs.aws.amazon.com/sdk-for-java/v2/developer-guide/client-configuration-http.html]
*Re-write*
The vast majority of the work has been re-namespacing and re-writing the AWS
object constructors into the builder pattern that AWS SDK 2.x is now using.
*KinesisClient*
>From the POV of how the KinesisClient is constructed and properties set, there
>has been a rather large change to the SDK.
The KinesisClient now takes multiple configurations (all builders), rather than
a single POJO style configuration class. I don't think Jackson is going to be
able to handle populating these new builders as nicely as the old single POJO.
[https://github.com/SaleCycle/flink/blob/FLINK-12847/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/util/AWSUtil.java#L89]
[https://github.com/SaleCycle/flink/blob/FLINK-12847/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/util/AWSUtil.java#L309]
*DynamoDBStreamsProxy*
I've re-coded the DynamoDBStreamsProxy to no longer use the
dynamodb-streams-kinesis-adapter because while there is a PR to update the
license for that project, there isn't one for updating to KCL/SDK v2.x
I've got at least one sticking point for that: what to set as the data property
of the record when it's converted. And there aren't any tests for that class
that I can see, so I'll need to write up a battery for it.
[https://github.com/SaleCycle/flink/blob/FLINK-12847/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/proxy/DynamoDBStreamsProxy.java#L309]
*Exceptions*
The possible Kinesis Exceptions have changed to something that has a better
hierarchy than the SDK v1.x exceptions, and has let me remove some of the code
around figuring out if an exception is retryable or not.
[https://github.com/SaleCycle/flink/blob/FLINK-12847/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/proxy/KinesisProxy.java#L453]
*Region Validation*
Another thing of note is that I can't find a properly suitable way to
"validate" a region in SDK v2.x. The new Region class looks to allow you to
create arbitrary regions. In this way you aren't locked to the regions that
existed at the time the library was released:
[https://github.com/aws/aws-sdk-java-v2/blob/41771af96e772dd85ff0ee159382a6de9b430515/codegen-lite/src/test/resources/software/amazon/awssdk/codegen/lite/regions/regions.java]
I've updated the "isValidRegion" method to still validate against the default
list in the Region class only to keep compatibility.
[https://github.com/SaleCycle/flink/blob/cef826fbd082c5e159d62eb5ccf99c7a44f54524/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/util/AWSUtil.java#L286]
*Big Questions To Answer*
1) Preference for populating the new builder configs
2) Setting the data property when converting a DynamoDB Record to a Kinesis
Record
or waiting to see if dynamodb-streams-kinesis-adapter is updated to v2.x
or looking at a larger re-write that doesn't try to merge the DynamoDB streams
namespace into the Kinesis streams namespace (they really don't overlap except
at the highest levels of the SDK)
> Update Kinesis Connectors to latest Apache licensed libraries
> -------------------------------------------------------------
>
> Key: FLINK-12847
> URL: https://issues.apache.org/jira/browse/FLINK-12847
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kinesis
> Reporter: Dyana Rose
> Assignee: Dyana Rose
> Priority: Major
>
> Currently the referenced Kinesis Client Library and Kinesis Producer Library
> code in the flink-connector-kinesis is licensed under the Amazon Software
> License which is not compatible with the Apache License. This then requires a
> fair amount of work in the CI pipeline and for users who want to use the
> flink-connector-kinesis.
> The Kinesis Client Library v2.x and the AWS Java SDK v2.x both are now on the
> Apache 2.0 license.
> [https://github.com/awslabs/amazon-kinesis-client/blob/master/LICENSE.txt]
> [https://github.com/aws/aws-sdk-java-v2/blob/master/LICENSE.txt]
> There is a PR for the Kinesis Producer Library to update it to the Apache 2.0
> license ([https://github.com/awslabs/amazon-kinesis-producer/pull/256])
> The task should include, but not limited to, upgrading KCL/KPL to new
> versions of Apache 2.0 license, changing licenses and NOTICE files in
> flink-connector-kinesis, and adding flink-connector-kinesis to build, CI and
> artifact publishing pipeline, updating the build profiles, updating
> documentation that references the license incompatibility
> The expected outcome of this issue is that the flink-connector-kinesis will
> be included with the standard build artifacts and will no longer need to be
> built separately by users.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
