rangareddy opened a new pull request, #19800: URL: https://github.com/apache/hudi/pull/19800
### Describe the issue this Pull Request addresses Closes #16374 (JIRA: [HUDI-7321](https://issues.apache.org/jira/browse/HUDI-7321)). Building the utilities bundle with `-Putilities-bundle-shade-hive` produces a jar that fails at runtime with: ``` Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat ``` **Root cause.** `hudi-hadoop-mr` is in the bundle's shade whitelist (`packaging/hudi-utilities-bundle/pom.xml:106`), and `HoodieParquetInputFormatBase:49 extends MapredParquetInputFormat`, which lives in `hive-exec`. The bundle declared five hive artifacts - `hive-service`, `hive-service-rpc`, `hive-jdbc`, `hive-metastore`, `hive-common` - and whitelisted the same five. **`hive-exec` was in neither list**, and its managed scope is `provided`, which never propagates transitively, so nothing could pull it in. The bundle therefore shipped subclasses whose superclass was absent. The reporter's workaround, "adding hive-exec resolved it", is the correct diagnosis. ### Summary and Changelog One file, `packaging/hudi-utilities-bundle/pom.xml`, +27: - Declares `hive-exec` at `${hive.exec.classifier}` with `scope=${utilities.bundle.hive.scope}`, matching the hive artifacts alongside it. The `core` classifier is what the rest of the build uses: it carries Hive's own classes without Hive's shaded copies of third-party libraries, so it does not drag protobuf, guava and friends into the bundle (10.3 MB rather than 44.2 MB). - Adds `<include>org.apache.hive:hive-exec</include>` to the shade whitelist. Scope defaults to `provided` (root pom), so **the default bundle is unchanged** and `hive-exec` is packaged only under `-Putilities-bundle-shade-hive`. The new dependency deliberately carries **no local `<exclusions>`**. The root pom's `dependencyManagement` entry for `hive-exec` already excludes `javax.mail`, the jetty aggregate, pentaho, log4j 1.x, log4j2, `slf4j-log4j12` and hbase, several of which the enforcer bans outright. Declaring exclusions locally *replaces* that managed set rather than adding to it, so an innocuous-looking local exclusion block would quietly let the banned artifacts back in. ### Verification Built the bundle both ways, then loaded the class from the jar with a **Hadoop-only classpath** - which is what a cluster running this bundle provides, and which is why the bundle has to carry hive itself: | | `MapredParquetInputFormat` | `HoodieParquetInputFormat` | | --- | --- | --- | | before, `-Putilities-bundle-shade-hive` | `ClassNotFoundException` | `NoClassDefFoundError: org/apache/hadoop/hive/ql/io/parquet/MapredParquetInputFormat` | | after, `-Putilities-bundle-shade-hive` | resolves | resolves | The "before" row is the reported failure, reproduced. Jar contents and sizes: ``` baseline (master), -Putilities-bundle-shade-hive 101.4 MB MapredParquetInputFormat: 0 entries fixed, -Putilities-bundle-shade-hive 111.6 MB MapredParquetInputFormat: 1 entry fixed, default build (no profile) 90.4 MB MapredParquetInputFormat: 0 entries ``` `MapredParquetInputFormat` lands under its original name; nothing is relocated under `org.apache.hudi.` (0 entries under `org/apache/hudi/org/apache/hadoop/hive/ql/`), so Hive's own loading of `HoodieParquetInputFormat` is unaffected. Full build with **checkstyle, RAT and enforcer enabled** passes (`BannedDependencies passed`), and the shaded jar contains nothing under `org/apache/log4j`, `org/apache/logging/log4j`, `org/slf4j/impl`, `org/apache/hadoop/hbase`, `org/pentaho` or `javax/mail`. ### Impact Only the opt-in `-Putilities-bundle-shade-hive` build changes: that bundle grows by ~10 MB and becomes self-contained for the Hive input formats it already ships. The default `hudi-utilities-bundle` is byte-for-byte unaffected, since `provided` scope keeps `hive-exec` out and the whitelist entry has nothing to match. Because `org.apache.hadoop.hive.ql.` is not in this bundle's relocation list, the profile's bundle now carries Hive `ql` classes under their original names. That is the existing semantics of this profile - it exists to put Hive inside the bundle - but it does mean the shaded jar can shadow a Hive install on a shared classpath. Relocating `hive.ql` instead was considered and rejected: it would rewrite `HoodieParquetInputFormatBase`'s superclass and break Hive's own loading of `HoodieParquetInputFormat`. ### Risk Level low - opt-in profile only, default bundle unchanged, verified against the enforcer. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable - [x] CI passes on my PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
