vinishjail97 opened a new pull request, #840:
URL: https://github.com/apache/incubator-xtable/pull/840
## What is the purpose of the pull request
Fixes #736. Running the bundled utilities jar failed with:
```
Exception in thread "main" java.lang.NoClassDefFoundError:
org/openjdk/jol/info/GraphLayout
at
org.apache.hudi.common.util.ObjectSizeCalculator.getObjectSize(ObjectSizeCalculator.java:57)
...
at org.apache.xtable.utilities.RunSync.main(RunSync.java:282)
Caused by: java.lang.ClassNotFoundException: org.openjdk.jol.info.GraphLayout
```
Hudi's `ObjectSizeCalculator` loads `org.openjdk.jol` at runtime
(metadata-table
reads go through it), but `xtable-utilities` declared `jol-core` with `test`
scope — inherited from the parent `dependencyManagement` — and never listed
it
in the shade `artifactSet`. So the `*-bundled.jar` was assembled without the
class, and any real sync that touches the Hudi metadata table blew up.
## Brief change log
- `xtable-utilities/pom.xml`: override `jol-core` to `runtime` scope with the
managed `${jol.core.version}` (mirroring how `xtable-service` already
declares
it), and add `org.openjdk.jol:jol-core` to the shade `<includes>`
allowlist so
it is actually bundled.
## Why this only surfaces when building the jar yourself
`xtable-utilities` is **not published to Maven and is excluded from the ASF
release process** — see the dev@ thread *"[DISCUSS] Excluding
xtable-utilities
from the release process"* and #536. It is an unshaded, un-relocated ~1GB fat
jar that bundles the full transitive closure of Spark/Hadoop/Hive/etc.; that
form can't be released under ASF licensing/release guidelines and isn't
practical to drop into an existing Spark job. Because there is no published
artifact, users build the bundle locally (as the reporter did), which is
exactly
the path where the missing `test`-scoped dependency shows up. This PR makes
that
local build produce a working jar again.
## The maintained path going forward
The long-term replacement for the fat utilities jar is
**`xtable-spark-runtime`**
(#836): a thin, relocated, drop-in Spark bundle activated by config only,
that
runs incremental `ConversionController.sync(...)` in-job — no separate
`RunSync`
process and no ~1GB unshaded jar. Implementation is in progress in #838
(packaging / thin bundle) and #839 (the `XTableSyncListener`). This PR is a
targeted fix so the existing utilities jar keeps working in the meantime; it
is
not a substitute for that effort.
## Verify this pull request
`jol-core` now resolves as a `runtime` dependency of the module:
```
$ mvn -o -pl xtable-utilities dependency:tree
-Dincludes=org.openjdk.jol:jol-core
[INFO] \- org.openjdk.jol:jol-core:jar:0.16:runtime
```
and its coordinate is in the shade allowlist, so it lands in the
`xtable-utilities_2.12-*-bundled.jar`. Running `RunSync` against a Hudi
source
that reads the metadata table no longer throws `NoClassDefFoundError`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
