[GitHub] flink issue #5663: [FLINK-8888] [Kinesis Connectors] Update the AWS SDK for ...
Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/5663 Travis CI pending after rebasing this, will push as soon at it gives us a green light... ---
[GitHub] flink issue #5663: [FLINK-8888] [Kinesis Connectors] Update the AWS SDK for ...
Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/5663 Thanks for checking this out. Merging to 1.5 and 1.6 then... ---
[GitHub] flink issue #5663: [FLINK-8888] [Kinesis Connectors] Update the AWS SDK for ...
Github user tweise commented on the issue: https://github.com/apache/flink/pull/5663 @StephanEwen we are looking to utilize this patch to override discovery using ListShards (initially in our connector extension). I have tested the SDK version change on top of our Flink 1.4 + patches branch with my kinesis quick check app and it works as expected. I believe @kailashhd had already addressed all other concerns? ---
[GitHub] flink issue #5663: [FLINK-8888] [Kinesis Connectors] Update the AWS SDK for ...
Github user kailashhd commented on the issue: https://github.com/apache/flink/pull/5663 Sorry for the delay in this. Confirming that I tested this using kinesalite and kinesis for both the consumer and producer functionality. I had some trouble when connecting this to kinesalite due to the following issue for consumers [FLINK-8936]. I am planning on fixing this in the next iteration and did not want for cleaner modular PRs. ---
[GitHub] flink issue #5663: [FLINK-8888] [Kinesis Connectors] Update the AWS SDK for ...
Github user kailashhd commented on the issue: https://github.com/apache/flink/pull/5663 Currently in flink connector we are depending only on aws-sdk-kinesis and not on aws-java-sdk-bundle and also don't depend on kinesisvideo. So by default the dependency on kinesisvideo is not included in the connector which means we don't have to exclude any dependencies. I also verified that there is no unwanted netty dependencies by running mvn dependency:tree. The only instance of netty is this: `[INFO] | +- org.apache.flink:flink-shaded-netty:jar:4.0.27.Final-2.0:provided` in accordance to the value in flink-parent pom. ---
[GitHub] flink issue #5663: [FLINK-8888] [Kinesis Connectors] Update the AWS SDK for ...
Github user bowenli86 commented on the issue: https://github.com/apache/flink/pull/5663 Is your testing Flink job both reading from and writing to Kinesis, aka both KCL and KPL are tested? If so, +1 ---
[GitHub] flink issue #5663: [FLINK-8888] [Kinesis Connectors] Update the AWS SDK for ...
Github user steveloughran commented on the issue: https://github.com/apache/flink/pull/5663 you don't need to shade it, just exclude it explicilty in your pom .it came in with kinesis-video, and only stops that feature from working if excluded [AWS 1488](https://github.com/aws/aws-sdk-java/issues/1488) ---
[GitHub] flink issue #5663: [FLINK-8888] [Kinesis Connectors] Update the AWS SDK for ...
Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/5663 I think the above means we need to check and either manually shade netty or, preferably, exclude the netty dependency (if it is not actually needed for Kinesis) ---
[GitHub] flink issue #5663: [FLINK-8888] [Kinesis Connectors] Update the AWS SDK for ...
Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/5663 Forwarding comment from JIRA from @steveloughran > If you are pulling in the shaded SDK, note that it's been declaring its netty dependencies unshaded: HADOOP-15264 ---
[GitHub] flink issue #5663: [FLINK-8888] [Kinesis Connectors] Update the AWS SDK for ...
Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/5663 Looks reasonable. Would be great if another Kinesis user could second this change, just to be safe. @tweise do you see any issues with upgrading AWS SDK here? ---
