junyuc25 commented on code in PR #44211:
URL: https://github.com/apache/spark/pull/44211#discussion_r1422082731
##########
connector/kinesis-asl/pom.xml:
##########
@@ -54,14 +54,38 @@
<scope>test</scope>
</dependency>
<dependency>
- <groupId>com.amazonaws</groupId>
+ <groupId>software.amazon.kinesis</groupId>
<artifactId>amazon-kinesis-client</artifactId>
Review Comment:
Hi @pan3793, KCL (software.amazon.kinesis:amazon-kinesis-client) is built on
top of Kinesis Data Stream API (software.amazon.awssdk:kinesis), and KCL
provides additional functionalities such as load balancing, error recovery etc.
According to this doc, generally it is recommended to use KCL over Kinesis Data
Stream
API:https://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-sdk.html
.
Perhaps we could try to raise an issue to the [KCL
repo](https://github.com/awslabs/amazon-kinesis-client) and ask if they can
release a "clean" version of the library with all the third-party dependencies
shaded. But currently I'm not aware there is such a hygienic version of KCL. I
guess we probably have to live with this issue if we use KCL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
