Jiyoung Yoo created IMPALA-15722:
------------------------------------
Summary: Optimize impala-minimal-s3a-aws-sdk JAR dependencies
after upgrade to Hadoop 3.4.x
Key: IMPALA-15722
URL: https://issues.apache.org/jira/browse/IMPALA-15722
Project: IMPALA
Issue Type: Improvement
Reporter: Jiyoung Yoo
The {{impala-minimal-s3a-aws-sdk}} module
({{{}java/shaded-deps/s3a-aws-sdk/pom.xml{}}}) is designed to produce a
lightweight shaded JAR containing only essential AWS SDK classes, keeping
Impala Docker images and distribution packages minimal.
As Impala builds against recent Hadoop versions (Hadoop 3.4.x), the underlying
{{hadoop-aws}} module has transitioned to AWS SDK v2
({{{}software.amazon.awssdk:bundle{}}}). However, current build configurations
still only target AWS SDK v1:
# {{java/executor-deps/pom.xml}} (and {{{}fe/pom.xml{}}}) only excludes
{{com.amazonaws:aws-java-sdk-bundle}} under {{{}hadoop-cloud-storage{}}},
missing an exclusion rule for {{{}software.amazon.awssdk:bundle{}}}.
# As a result, the full unshaded AWS SDK v2 bundle ({{{}bundle-2.49.6.jar{}}},
~686 MB) is pulled in as a transitive dependency and copied directly into
{{fe/target/dependency/.}}
# {{java/shaded-deps/s3a-aws-sdk/pom.xml}} lacks the shading configuration and
exclusion filters for AWS SDK v2 services.
h4. Dependency Tree Output
Running {{mvn dependency:tree}} under {{fe/}} reveals that the full AWS SDK v2
bundle is currently being included:
[INFO] | +-
org.apache.hadoop:hadoop-cloud-storage:jar:3.4.2.7.3.2.20000-221:compile
[INFO] | | +- org.apache.hadoop:hadoop-aws:jar:3.4.2.7.3.2.20000-221:compile
[INFO] | | | +- software.amazon.awssdk:{*}bundle{*}{*}:jar:{*}2.49.6:compile
INFO] | | | +-
software.amazon.s3.analyticsaccelerator:analyticsaccelerator-s3:jar:1.2.1:compile
INFO] | | | \-
org.wildfly.openssl:wildfly-openssl:jar:2.2.5.ClouderaMoreFinal:compile
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
