This is an automated email from the ASF dual-hosted git repository.
vinishjail97 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-xtable.git
The following commit(s) were added to refs/heads/main by this push:
new 594976be [896] Shrink the xtable-utilities bundled jar from 1.1 GB to
423 MB (#897)
594976be is described below
commit 594976bed7b619e0b91cabd4aae9f6be26da17f0
Author: Sylwester Lachiewicz <[email protected]>
AuthorDate: Fri Sep 18 01:00:19 2026 +0200
[896] Shrink the xtable-utilities bundled jar from 1.1 GB to 423 MB (#897)
* Bundle only the AWS SDK artifacts S3A uses
xtable-utilities depended on software.amazon.awssdk:bundle, the
all-services jar. It was 1957 MB of the 3094 MB shaded jar, across
379410 files.
hadoop-aws 3.4.1 S3A references only services/s3, plus STSClient for
assumed roles, and the transfer, http, core, awscore and auth packages.
No other AWS service appears anywhere in its 418 classes. So: s3, sts,
netty-nio-client and s3-transfer-manager. Glue keeps coming granularly
via xtable-aws.
The root pom now imports the AWS BOM instead of managing artifacts one
by one.
Bundled jar 1098 MB -> 473 MB, 582980 entries -> 212508.
Not verified against a live S3 endpoint. A credentials provider or
signer configured by class name could still reach outside this set.
* Drop RocksDB natives for platforms we do not run on
ppc64le, s390x and 32-bit Linux, 84 MB across six files. x86_64 and
aarch64 keep their musl builds, macOS and Windows stay for development.
The excludes go in the root pom's shade execution, not the module: an
execution-level <filters> replaces a plugin-level one rather than
merging with it, so a filter added in the module is silently dropped.
Bundled jar 473 MB -> 443 MB.
* Drop grpc-xds and groovy from the utilities bundle
grpc-xds is only reachable through xds:// targets, which nothing here
uses; its 40 MB is referenced only by gRPC's own code. groovy-all 2.4.11
arrives via Hive and nothing outside Groovy touches it.
Bundled jar 443 MB -> 423 MB.
* Whitelist the AWS artifacts s3 and sts pull in
arns, aws-query-protocol, aws-xml-protocol and crt-core were in the old
bundle but not in the shade includes, so the S3 and STS clients would
have failed at runtime. Found with jdeps; the earlier constant-pool scan
only covered what S3A names directly, not what those classes need.
hadoop-aws also brings bundle 2.24.6 transitively, which the direct
2.29.40 declaration had been masking. Excluded so it cannot return.
* Keep the relocated HTTP client hadoop-aws links against, and grpc-xds
hadoop-aws 3.4.1 is compiled against the AWS bundle, so STSClientFactory
and ConfigureShadedAWSSocketFactory reference the relocated
software.amazon.awssdk.thirdparty.org.apache.http client and call
ApacheHttpClient.Builder.socketFactory with that relocated type. The
standalone apache-client has the unrelocated signature, so setting
fs.s3a.assumed.role.sts.endpoint threw NoClassDefFoundError and
fs.s3a.ssl.channel.mode was silently ignored.
Take from bundle only the relocated client, the SDK adapter bound to it
and the relocated logging and codec classes they call, at the same SDK
version as the standalone artifacts so no class is shipped twice.
grpc-xds comes back because google-c2p delegates to it on Google Cloud.
---
pom.xml | 30 ++++++++++++++++++++----
xtable-utilities/pom.xml | 61 ++++++++++++++++++++++++++++++++++++++++++++++--
2 files changed, 85 insertions(+), 6 deletions(-)
diff --git a/pom.xml b/pom.xml
index d585f44f..ee3676ce 100644
--- a/pom.xml
+++ b/pom.xml
@@ -448,17 +448,19 @@
<scope>runtime</scope>
</dependency>
- <!-- AWS -->
+ <!-- AWS. The BOM manages every SDK artifact except bundle, so
modules name only what they use. -->
<dependency>
<groupId>software.amazon.awssdk</groupId>
- <artifactId>bundle</artifactId>
+ <artifactId>bom</artifactId>
<version>${aws.version}</version>
- <scope>runtime</scope>
+ <type>pom</type>
+ <scope>import</scope>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
- <artifactId>glue</artifactId>
+ <artifactId>bundle</artifactId>
<version>${aws.version}</version>
+ <scope>runtime</scope>
</dependency>
<!-- Protobuf -->
@@ -674,8 +676,28 @@
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
<exclude>**/Log4j2Plugins.dat</exclude>
+ <!-- RocksDB natives for platforms
we do not run on. -->
+
<exclude>librocksdbjni-linux-ppc64le*</exclude>
+
<exclude>librocksdbjni-linux-s390x*</exclude>
+
<exclude>librocksdbjni-linux32*</exclude>
</excludes>
</filter>
+ <!-- hadoop-aws is compiled against the
bundle, so its STS endpoint and
+ SSL channel code links to the
relocated Apache HTTP client. Take
+ that client, the SDK adapter bound to
it and the relocated logging
+ and codec classes they call; every
other SDK class comes from the
+ standalone artifact of the same
version. -->
+ <filter>
+
<artifact>software.amazon.awssdk:bundle</artifact>
+ <includes>
+
<include>software/amazon/awssdk/http/apache/**</include>
+
<include>software/amazon/awssdk/thirdparty/org/apache/http/**</include>
+
<include>software/amazon/awssdk/thirdparty/org/apache/commons/codec/**</include>
+
<include>software/amazon/awssdk/thirdparty/org/apache/commons/logging/**</include>
+
<include>software/amazon/awssdk/thirdparty/org/slf4j/**</include>
+
<include>META-INF/services/software.amazon.awssdk.http.SdkHttpService</include>
+ </includes>
+ </filter>
</filters>
<shadedArtifactAttached>true</shadedArtifactAttached>
<shadedClassifierName>bundled</shadedClassifierName>
diff --git a/xtable-utilities/pom.xml b/xtable-utilities/pom.xml
index af69e688..dc8ff7b6 100644
--- a/xtable-utilities/pom.xml
+++ b/xtable-utilities/pom.xml
@@ -221,7 +221,33 @@
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-aws</artifactId>
+ <exclusions>
+ <!-- The all-services SDK jar. We name the artifacts S3A uses
instead. -->
+ <exclusion>
+ <groupId>software.amazon.awssdk</groupId>
+ <artifactId>bundle</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <!-- What hadoop-aws S3A needs. The all-services `bundle` artifact was
63% of the shaded jar. -->
+ <dependency>
+ <groupId>software.amazon.awssdk</groupId>
+ <artifactId>s3</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>software.amazon.awssdk</groupId>
+ <artifactId>sts</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>software.amazon.awssdk</groupId>
+ <artifactId>netty-nio-client</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>software.amazon.awssdk</groupId>
+ <artifactId>s3-transfer-manager</artifactId>
</dependency>
+ <!-- Only for the relocated Apache HTTP client hadoop-aws links
against; the shade
+ filter in the root pom drops everything else in it. -->
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>bundle</artifactId>
@@ -440,6 +466,7 @@
<include>io.grpc:grpc-services</include>
<include>io.grpc:grpc-stub</include>
<include>io.grpc:grpc-util</include>
+ <!-- google-c2p delegates to xds on Google Cloud.
-->
<include>io.grpc:grpc-xds</include>
<include>io.javalin:javalin</include>
<include>io.netty:netty</include>
@@ -665,7 +692,6 @@
<include>org.bouncycastle:bcprov-jdk18on</include>
<include>org.checkerframework:checker-compat-qual</include>
<include>org.checkerframework:checker-qual</include>
- <include>org.codehaus.groovy:groovy-all</include>
<include>org.codehaus.jackson:jackson-core-asl</include>
<include>org.codehaus.jackson:jackson-jaxrs</include>
<include>org.codehaus.jackson:jackson-mapper-asl</include>
@@ -746,8 +772,39 @@
<include>org.xerial.snappy:snappy-java</include>
<include>org.yaml:snakeyaml</include>
<include>oro:oro</include>
- <!-- Use only the AWS SDK bundle to avoid mixing
incompatible SDK classes. -->
+
<include>software.amazon.awssdk:annotations</include>
<include>software.amazon.awssdk:bundle</include>
+ <include>software.amazon.awssdk:auth</include>
+ <include>software.amazon.awssdk:arns</include>
+
<include>software.amazon.awssdk:aws-query-protocol</include>
+
<include>software.amazon.awssdk:aws-xml-protocol</include>
+ <include>software.amazon.awssdk:crt-core</include>
+ <include>software.amazon.awssdk:aws-core</include>
+
<include>software.amazon.awssdk:aws-json-protocol</include>
+ <include>software.amazon.awssdk:s3</include>
+ <include>software.amazon.awssdk:sts</include>
+
<include>software.amazon.awssdk:netty-nio-client</include>
+
<include>software.amazon.awssdk:s3-transfer-manager</include>
+ <include>software.amazon.awssdk:checksums</include>
+
<include>software.amazon.awssdk:checksums-spi</include>
+
<include>software.amazon.awssdk:endpoints-spi</include>
+ <include>software.amazon.awssdk:glue</include>
+ <include>software.amazon.awssdk:http-auth</include>
+
<include>software.amazon.awssdk:http-auth-aws</include>
+
<include>software.amazon.awssdk:http-auth-aws-eventstream</include>
+
<include>software.amazon.awssdk:http-auth-spi</include>
+
<include>software.amazon.awssdk:http-client-spi</include>
+
<include>software.amazon.awssdk:identity-spi</include>
+
<include>software.amazon.awssdk:json-utils</include>
+
<include>software.amazon.awssdk:metrics-spi</include>
+ <include>software.amazon.awssdk:profiles</include>
+
<include>software.amazon.awssdk:protocol-core</include>
+ <include>software.amazon.awssdk:regions</include>
+ <include>software.amazon.awssdk:retries</include>
+
<include>software.amazon.awssdk:retries-spi</include>
+ <include>software.amazon.awssdk:sdk-core</include>
+
<include>software.amazon.awssdk:third-party-jackson-core</include>
+ <include>software.amazon.awssdk:utils</include>
<include>software.amazon.eventstream:eventstream</include>
<include>stax:stax-api</include>
<include>tomcat:jasper-compiler</include>
