[
https://issues.apache.org/jira/browse/FLINK-7673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16177962#comment-16177962
]
Tzu-Li (Gordon) Tai edited comment on FLINK-7673 at 9/23/17 8:21 PM:
---------------------------------------------------------------------
Yes, it would definitely be nice to eventually have a ASL license-free Kinesis
connector that we can publish.
If we were to reverse engineer the functionality that KCL / KPL provides, the
amount of work would vary across the consumer and producer:
- For the consumer, we would only need to work out how to perform the record
message aggregation. I think KCL uses protobuf to do this magic. It might
simply be a matter of migrating some protobuf generated classes to the Kinesis
connector.
- For the producer, I previously already worked on a ASL-free version here:
https://github.com/tzulitai/flink/tree/FLINK-3229-RemoveKclKplRework. Its
functionality probably isn't on-par with the current producer which uses KPL,
so we would need to look into that. Robert and I initially did not decide to
use it because we wanted to wait and see if there may be some possibilities of
license changes on the AWS side.
Either way, I would be very happy to see some progress on this front. It would
make our AWS users much more happier :)
was (Author: tzulitai):
Yes, it would definitely be nice to eventually have a ASL license-free Kinesis
connector that we can publish.
If we were to reverse engineer this, the amount of work would vary across the
consumer and producer:
- For the consumer, we would only need to work out how to perform the record
message aggregation. I think KCL uses protobuf to do this magic. It might
simply be a matter of migrating some protobuf generated classes to the Kinesis
connector.
- For the producer, I previously already worked on a ASL-free version here:
https://github.com/tzulitai/flink/tree/FLINK-3229-RemoveKclKplRework. Its
functionality probably isn't on-par with the current producer which uses KPL,
so we would need to look into that. Robert and I initially did not decide to
use it because we wanted to wait and see if there may be some possibilities of
license changes on the AWS side.
Either way, I would be very happy to see some progress on this front. It would
make our AWS users much more happier :)
> Flink Kinesis connector - remove dependency on the ASL code
> -----------------------------------------------------------
>
> Key: FLINK-7673
> URL: https://issues.apache.org/jira/browse/FLINK-7673
> Project: Flink
> Issue Type: Improvement
> Components: Kinesis Connector
> Reporter: Radoslaw Gruchalski
>
> Currently the Flink Kinesis connector depends on two artifacts which are
> available under Amazon Software License code. The artifacts are:
> {noformat}
> <dependency>
> <groupId>com.amazonaws</groupId>
> <artifactId>amazon-kinesis-producer</artifactId>
> <version>${aws.kinesis-kpl.version}</version>
> </dependency>
> <dependency>
> <groupId>com.amazonaws</groupId>
> <artifactId>amazon-kinesis-client</artifactId>
> <version>${aws.kinesis-kcl.version}</version>
> <!--
> We're excluding the below from the KCL since
> we'll only be using the
>
> com.amazonaws.services.kinesis.clientlibrary.types.UserRecord class, which
> will not need these dependencies.
> -->
> <exclusions>
> <exclusion>
> <groupId>com.amazonaws</groupId>
>
> <artifactId>aws-java-sdk-dynamodb</artifactId>
> </exclusion>
> <exclusion>
> <groupId>com.amazonaws</groupId>
>
> <artifactId>aws-java-sdk-cloudwatch</artifactId>
> </exclusion>
> </exclusions>
> </dependency>
> {noformat}
> This prevents the connector being published to Maven Central. I would like to
> contribute to Flink Kinesis connector by replacing the ASL licensed code with
> an implementation using AWS SDK for Java. For that, I'd like to understand
> two things:
> - what's the correct process for making the contribution (contributor
> agreement, etc.)
> - what's the minimum functionality the alternative implementation would have
> to provide in order to get accepted
> There are three classes in the connector's source requiring modification:
> - org.apache.flink.streaming.connectors.kinesis.FlinkKinesisProducer
> - org.apache.flink.streaming.connectors.kinesis.util.KinesisConfigUtil
> - org.apache.flink.streaming.connectors.kinesis.internals.ShardConsumer
> Thank you.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
