vrozov commented on code in PR #52654:
URL: https://github.com/apache/spark/pull/52654#discussion_r2456067043
##########
connector/kinesis-asl/pom.xml:
##########
@@ -64,10 +64,38 @@
<version>${aws.java.sdk.version}</version>
</dependency>
<dependency>
- <groupId>com.amazonaws</groupId>
+ <groupId>software.amazon.kinesis</groupId>
<artifactId>amazon-kinesis-producer</artifactId>
<version>${aws.kinesis.producer.version}</version>
<scope>test</scope>
+ <exclusions>
+ <!-- auth and sts are excluded for SBT dependency management, so those
dependencies
+ can be added explicitly -->
+ <exclusion>
+ <groupId>software.amazon.awssdk</groupId>
+ <artifactId>auth</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>software.amazon.awssdk</groupId>
+ <artifactId>sts</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.kjetland</groupId>
+ <artifactId>mbknor-jackson-jsonschema_2.12</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>software.amazon.awssdk</groupId>
+ <artifactId>auth</artifactId>
+ <version>${aws.java.sdk.v2.version}</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>software.amazon.awssdk</groupId>
+ <artifactId>sts</artifactId>
+ <version>${aws.java.sdk.v2.version}</version>
+ <scope>test</scope>
Review Comment:
@sarutak @dongjoon-hyun The issue is caused by `4.0` dependency on
`hadoop-aws:3.4.1` that was upgraded to `3.4.2` in `4.1`. `hadoop-aws:3.4.1`
has transitive dependency on AWS Java SDK v2 `2.24.6` that is not compatible
with Kinesis Producer transitive dependency on AWS Java SDK v2 `2.29.24`. In
general, Spark should be enforcing specific version of AWS Java SDK v2 (the one
specified for `aws.java.sdk.v2.version`) instead of relying on transitive
dependencies that may not be compatible with each other.
Note that this is not an issue for maven build as it correctly handles
transitive dependencies, and causes the issue that you noticed in SBT.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
