slachiewicz opened a new issue, #896: URL: https://github.com/apache/incubator-xtable/issues/896
### Feature Request / Improvement `xtable-utilities` is 795 lines of Java in two classes, `RunSync` and `RunCatalogSync`. It builds a 1.1 GB bundled jar: 3.1 GB uncompressed, 582,980 entries, 483 whitelist entries. Measured by top-level package: | | MB uncompressed | files | |---|---:|---:| | `software/amazon/awssdk` | 1957 | 379,410 | | `org/apache/hadoop` | 258 | 53,612 | | `org/apache/spark` | 91 | 12,129 | | `org/apache/paimon` | 86 | 18,840 | | RocksDB natives | 159 | 13 | | `io/grpc/xds` | 40 | 4,805 | | `it/unimi/dsi` | 39 | 10,665 | | `org/apache/hudi` | 17 | 2,890 | | `org/apache/iceberg` | 12 | 3,126 | The formats XTable exists to translate are about 1% of the jar. This is the "reduce the size of bundled jar" step from the 2024 [DISCUSS] thread on excluding `xtable-utilities` from the release, where @vinothchandar suggested moving to the Hudi model of whitelisting what compute engines do not already provide. The whitelist has grown rather than shrunk since. ### What can be cut Four changes, no source touched, all measured: 1. `software.amazon.awssdk:bundle` is the all-services jar. `hadoop-aws` 3.4.1's S3A references exactly one service package, `services/s3`, plus `STSClient`, and the `transfer`, `http`, `core`, `awscore` and `auth` packages. Nothing else. Verified by scanning the constant pools of all 418 `org.apache.hadoop.fs.s3a` classes. Replacing `bundle` with `s3`, `sts`, `netty-nio-client` and `s3-transfer-manager` takes the SDK from 1957 MB to 50 MB. 2. RocksDB ships a native library per platform. Dropping ppc64le, s390x and 32-bit Linux removes 84 MB. x86_64 and aarch64 keep their musl builds; macOS and Windows stay for development. 3. `io.grpc:grpc-xds` is 40 MB, referenced only from gRPC's own code, and reachable only through `xds://` targets. 4. `org.codehaus.groovy:groovy-all` 2.4.11 arrives via Hive; nothing outside Groovy references it. Together: **1098 MB to 423 MB**, entries 582,980 to 203,041. ### What cannot be cut without a design change Spark's 91 MB is load-bearing today: `xtable-core`'s Delta support uses `delta-core`, a Spark library, across 12 classes (`SparkSession`, `DeltaLog`, `StructType`). `delta-kernel-api` and `delta-kernel-defaults` are already dependencies at 4.0.0. Kernel is the Spark-free Delta API, so migrating that code would let Spark and much of its transitive weight go. That is a larger piece of work and relates to #611. `hadoop-azure` and the GCS connector have no static referrer, but they are `ServiceLoader`-loaded filesystems, so their absence would be a lost feature rather than dead weight. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! *This issue was created with AI assistance.* -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
