This is an automated email from the ASF dual-hosted git repository.
vinishjail97 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-xtable.git
The following commit(s) were added to refs/heads/main by this push:
new 4daec279 [736] Make the bundled xtable-utilities jar runnable again
(#840)
4daec279 is described below
commit 4daec2795e6ec1234f0d0c8aa05aadee0780b714
Author: Vinish Reddy <[email protected]>
AuthorDate: Wed Jul 8 11:28:37 2026 +0530
[736] Make the bundled xtable-utilities jar runnable again (#840)
Running the bundled utilities jar the way the docs describe
(java -jar xtable-utilities_2.12-*-bundled.jar --datasetConfig config.yaml)
failed on a chain of NoClassDefFoundErrors, and syncing *to* Hudi was
silently skipped. Two upstream changes caused this: #822 replaced the
maven-shade "bundle everything" default with an explicit artifactSet
allowlist, and the Hudi 1.x bump split hudi-common into hudi-common +
hudi-io + hudi-hadoop-common.
Runtime dependencies dropped from the shaded jar:
org.slf4j:slf4j-api -> org/slf4j/LoggerFactory
org.apache.hudi:hudi-hadoop-common ->
org/apache/hudi/hadoop/fs/HadoopFSUtils
org.apache.hudi:hudi-io -> (runtime transitive, not
allowlisted)
org.openjdk.jol:jol-core -> org/openjdk/jol/info/GraphLayout
(#736)
Fixes:
- Add slf4j-api, hudi-io and jol-core to the shade allowlist.
- jol-core is pinned to test scope by the parent dependencyManagement, so
override it to runtime (Hudi's ObjectSizeCalculator loads it at runtime).
- hudi-hadoop-common is only a provided/test transitive, so declare it as a
runtime dependency of the standalone bundle and allowlist it.
- hudi-java-client was declared twice (compile + a redundant test-scoped
copy); Maven's last-declaration-wins downgraded it to test scope, so the
shade plugin excluded the Hudi write client (HoodieJavaEngineContext) and
RunSync skipped HudiConversionTarget. Remove the redundant test
declaration
so the compile one takes effect and the write client is bundled.
Verified end-to-end via java -jar against real tables:
- Hudi (table v9, metadata table enabled) -> Delta + Iceberg: reads the Hudi
metadata table (the ObjectSizeCalculator/jol path from this issue) and
reports "Sync is successful for the following formats ICEBERG,DELTA".
- Iceberg -> Hudi: writes a real Hudi table (.hoodie timeline with a
completed replacecommit) and reports "Sync is successful ... HUDI".
Closes #736.
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
---
xtable-utilities/pom.xml | 32 ++++++++++++++++++++++++++------
1 file changed, 26 insertions(+), 6 deletions(-)
diff --git a/xtable-utilities/pom.xml b/xtable-utilities/pom.xml
index 67142199..d8832abf 100644
--- a/xtable-utilities/pom.xml
+++ b/xtable-utilities/pom.xml
@@ -153,6 +153,20 @@
<groupId>org.apache.hudi</groupId>
<artifactId>hudi-common</artifactId>
</dependency>
+ <!--
+ Hudi 1.x split the former hudi-common into hudi-common + hudi-io +
+ hudi-hadoop-common. xtable-core compiles against these with the Hudi
+ Spark bundle in provided scope (a real Spark cluster supplies them),
+ but the standalone utilities jar has no such runtime, so it must
bundle
+ hudi-hadoop-common itself (hudi-io comes in transitively at runtime).
+ Without this, RunSync fails with NoClassDefFoundError:
+ org/apache/hudi/hadoop/fs/HadoopFSUtils.
+ -->
+ <dependency>
+ <groupId>org.apache.hudi</groupId>
+ <artifactId>hudi-hadoop-common</artifactId>
+ <version>${hudi.version}</version>
+ </dependency>
<dependency>
<groupId>org.apache.hudi</groupId>
<artifactId>hudi-java-client</artifactId>
@@ -232,15 +246,17 @@
<artifactId>hudi-spark${spark.version.prefix}-bundle_${scala.binary.version}</artifactId>
<scope>test</scope>
</dependency>
- <dependency>
- <groupId>org.apache.hudi</groupId>
- <artifactId>hudi-java-client</artifactId>
- <scope>test</scope>
- </dependency>
+ <!--
+ Hudi's ObjectSizeCalculator loads org.openjdk.jol at runtime (not
only in tests),
+ so the bundled utilities jar must ship it. The parent
dependencyManagement pins
+ jol-core to test scope, so override to runtime here and add it to
the shaded
+ artifactSet below. See
https://github.com/apache/incubator-xtable/issues/736.
+ -->
<dependency>
<groupId>org.openjdk.jol</groupId>
<artifactId>jol-core</artifactId>
- <scope>test</scope>
+ <version>${jol.core.version}</version>
+ <scope>runtime</scope>
</dependency>
</dependencies>
@@ -574,6 +590,8 @@
<include>org.apache.httpcomponents.core5:httpcore5-h2</include>
<include>org.apache.hudi:hudi-client-common</include>
<include>org.apache.hudi:hudi-common</include>
+
<include>org.apache.hudi:hudi-hadoop-common</include>
+ <include>org.apache.hudi:hudi-io</include>
<include>org.apache.hudi:hudi-java-client</include>
<include>org.apache.hudi:hudi-timeline-service</include>
<include>org.apache.iceberg:iceberg-api</include>
@@ -696,6 +714,7 @@
<include>org.json4s:json4s-scalap_${scala.binary.version}</include>
<include>org.lz4:lz4-java</include>
<include>org.objenesis:objenesis</include>
+ <include>org.openjdk.jol:jol-core</include>
<include>org.reactivestreams:reactive-streams</include>
<include>org.roaringbitmap:RoaringBitmap</include>
<include>org.roaringbitmap:shims</include>
@@ -707,6 +726,7 @@
<include>org.scala-lang.modules:scala-xml_${scala.binary.version}</include>
<include>org.slf4j:jcl-over-slf4j</include>
<include>org.slf4j:jul-to-slf4j</include>
+ <include>org.slf4j:slf4j-api</include>
<include>org.slf4j:slf4j-reload4j</include>
<include>org.threeten:threeten-extra</include>
<include>org.threeten:threetenbp</include>
