okumin commented on code in PR #6398:
URL: https://github.com/apache/hive/pull/6398#discussion_r3025254407
##########
pom.xml:
##########
@@ -271,10 +272,25 @@
<dependencyManagement>
<dependencies>
<!-- dependencies are always listed in sorted order by groupId,
artifactId -->
+ <dependency>
+ <groupId>software.amazon.awssdk</groupId>
+ <artifactId>bundle</artifactId>
+ <version>${aws-java-sdk.version}</version>
+ </dependency>
<dependency>
<groupId>com.amazonaws.secretsmanager</groupId>
<artifactId>aws-secretsmanager-caching-java</artifactId>
<version>${aws-secretsmanager-caching.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>software.amazon.awssdk</groupId>
+ <artifactId>aws-crt-client</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>software.amazon.awssdk</groupId>
+ <artifactId>secretsmanager</artifactId>
Review Comment:
Because a user is expected to bring their own bundle jar to the classpath.
AWS SDK has hierarchical dependencies. For example,
`software.amazon.awssdk/secretsmanager` depends on
`software.amazon.awssdk/http-auth-aws`, and
`software.amazon.awssdk/http-auth-aws` depends on
`software.amazon.awssdk/utils`. The versions of the AWS SDK family need to be
consistent, including any dependencies from S3A or elsewhere.
In a typical environment, most AWS users would use the bundle jar from
hadoop-aws or their own version.
```
root@acea28e120d3:/# ls /opt/hadoop/share/hadoop/tools/lib/ | grep bundle
bundle-2.24.6.jar
root@acea28e120d3:/# /opt/java/jdk11/bin/jar tf
/opt/hadoop/share/hadoop/tools/lib/bundle-2.24.6.jar | head
META-INF/
META-INF/MANIFEST.MF
META-INF/LICENSE.txt
META-INF/NOTICE.txt
META-INF/maven/
META-INF/maven/software.amazon.awssdk/
META-INF/maven/software.amazon.awssdk/bundle/
META-INF/maven/software.amazon.awssdk/bundle/pom.xml
META-INF/maven/software.amazon.awssdk/bundle/pom.properties
META-INF/maven/software.amazon.awssdk/bundle-sdk/
```
I'd say this change isn't surprising, since we didn't bundle the SDK in 4.2
or earlier versions.
https://github.com/apache/hive/blob/rel/release-4.2.0/pom.xml#L287-L296
##########
ql/pom.xml:
##########
@@ -166,6 +166,11 @@
<groupId>org.apache.hive</groupId>
<artifactId>hive-storage-api</artifactId>
</dependency>
+ <dependency>
+ <groupId>software.amazon.awssdk</groupId>
+ <artifactId>bundle</artifactId>
Review Comment:
Because we want to avoid `ClassNotFoundException` on
[AWSSecretsManagerSecretSourceTest](https://github.com/apache/hive/blob/rel/release-4.2.0/ql/src/test/org/apache/hadoop/hive/ql/secrets/AWSSecretsManagerSecretSourceTest.java).
```
Caused by: java.lang.NoClassDefFoundError:
software/amazon/awssdk/services/secretsmanager/SecretsManagerClient
at java.base/java.lang.Class.getDeclaredFields0(Native Method)
at java.base/java.lang.Class.privateGetDeclaredFields(Class.java:3475)
at java.base/java.lang.Class.getDeclaredFields(Class.java:2544)
at
net.bytebuddy.description.type.TypeDescription$ForLoadedType.getDeclaredFields(TypeDescription.java:8936)
at
net.bytebuddy.dynamic.scaffold.InstrumentedType$Factory$Default$1.represent(InstrumentedType.java:437)
at net.bytebuddy.ByteBuddy.redefine(ByteBuddy.java:923)
at net.bytebuddy.ByteBuddy.redefine(ByteBuddy.java:898)
at
org.mockito.internal.creation.bytebuddy.InlineBytecodeGenerator.transform(InlineBytecodeGenerator.java:396)
at
java.instrument/java.lang.instrument.ClassFileTransformer.transform(ClassFileTransformer.java:244)
at
java.instrument/sun.instrument.TransformerManager.transform(TransformerManager.java:188)
at
java.instrument/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:610)
at
java.instrument/sun.instrument.InstrumentationImpl.retransformClasses0(Native
Method)
at
java.instrument/sun.instrument.InstrumentationImpl.retransformClasses(InstrumentationImpl.java:225)
at
org.mockito.internal.creation.bytebuddy.InlineBytecodeGenerator.triggerRetransformation(InlineBytecodeGenerator.java:287)
... 42 more
Caused by: java.lang.ClassNotFoundException:
software.amazon.awssdk.services.secretsmanager.SecretsManagerClient
at
java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
at
java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
... 56 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
