deepakpanda93 commented on issue #15094:
URL: https://github.com/apache/hudi/issues/15094#issuecomment-5313328509
This issue was reviewed as part of the JIRA-migrated backlog triage.
Findings: fixed. This was a bundle packaging defect, verified by inspecting
the released artifacts rather than the source.
In `hudi-flink-bundle_2.11-0.10.1.jar` — the exact jar in the stack trace —
`HMSDDLExecutor` was compiled against the *relocated* name
`org/apache/hudi/org/apache/hadoop/hive/ql/metadata/Hive`, while the jar
contained zero classes under `org/apache/hudi/org/apache/hadoop/hive/`. The
shade relocation rewrote the references but the Hive classes were never
bundled, so the reference could not resolve regardless of what was on the
Flink classpath. That is the `NoClassDefFoundError` reported here.
From 0.14.2 onward the relocation is gone. Checked
`hudi-flink1.14-bundle-0.14.2`, `hudi-flink1.20-bundle-1.0.2` and
`hudi-flink1.20-bundle-1.2.0`: all reference the plain
`org/apache/hadoop/hive/ql/metadata/Hive`, and current master's
`packaging/hudi-flink-bundle/pom.xml` has no `org.apache.hadoop.hive`
relocation pattern at all.
Two notes for anyone who lands here from a search:
- The failure is in the **write** job's hive sync
(`StreamWriteOperatorCoordinator.syncHive` → `HiveSyncContext.hiveSyncTool` →
`HiveSyncTool.<init>`), not in reading the synced table, despite the title.
- A similar-looking error is still possible on 1.x, but it is a different
problem: `flink.bundle.hive.scope` defaults to `provided`, so Hive classes are
bundled only when building with `-Pflink-bundle-shade-hive3` (or
`-hive1`/`-hive2`). Missing Hive jars produce `NoClassDefFoundError` at the
*unrelocated* name, which is resolvable by fixing the classpath. If you hit
that, please open a new
issue rather than reopening this one.
PR #5188 was closed without merging; the duplicate `hive-exec` block it
targeted is still in the root pom, but it turned out not to be the cause — the
flink bundle sets its own scope via `${flink.bundle.hive.scope}`, which takes
precedence over dependencyManagement.
Closing as fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
