rangareddy commented on code in PR #19800:
URL: https://github.com/apache/hudi/pull/19800#discussion_r3921638496
##########
packaging/hudi-utilities-bundle/pom.xml:
##########
@@ -355,6 +357,31 @@
<scope>${utilities.bundle.hive.scope}</scope>
</dependency>
+ <!--
+ Supplies org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat,
which is the superclass of
Review Comment:
Yes, taken this route instead.
`get{InputFormat,OutputFormat,SerDe}ClassName` now return string literals, so
no Hive class loads at sync time and the packaging change is reverted in full.
That closes #16374 for the utilities, slim and spark bundles at once, with no
bundling and no size growth.
##########
packaging/hudi-utilities-bundle/pom.xml:
##########
@@ -355,6 +357,31 @@
<scope>${utilities.bundle.hive.scope}</scope>
</dependency>
+ <!--
+ Supplies org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat,
which is the superclass of
+ HoodieParquetInputFormatBase in hudi-hadoop-mr, which is whitelisted in
the shade artifactSet above. Without
+ this, the bundle ships those subclasses with their superclass missing
and any use of the Hive input formats fails
+ with ClassNotFoundException on MapredParquetInputFormat (HUDI-7321).
+
+ The core classifier is what the rest of the build uses
(hive.exec.classifier): it carries Hive's own
+ classes without Hive's shaded copies of third-party libraries, so it
does not drag protobuf, guava and
+ friends into this bundle. Scope follows utilities.bundle.hive.scope like
the hive artifacts above, so it
+ is provided by default and only packaged under
-Putilities-bundle-shade-hive.
+ -->
+ <dependency>
+ <groupId>${hive.groupid}</groupId>
+ <artifactId>hive-exec</artifactId>
Review Comment:
Confirmed and this is what decided it: the `core` jar has 0 `serde2/`
entries against 578 in the uber jar, so the bundling fix died one call later
exactly as you describe. The literal-constants change makes it moot, since
`getSerDeClassName` no longer loads `ParquetHiveSerDe` at all.
##########
packaging/hudi-utilities-bundle/pom.xml:
##########
@@ -355,6 +357,31 @@
<scope>${utilities.bundle.hive.scope}</scope>
</dependency>
+ <!--
+ Supplies org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat,
which is the superclass of
+ HoodieParquetInputFormatBase in hudi-hadoop-mr, which is whitelisted in
the shade artifactSet above. Without
+ this, the bundle ships those subclasses with their superclass missing
and any use of the Hive input formats fails
+ with ClassNotFoundException on MapredParquetInputFormat (HUDI-7321).
+
+ The core classifier is what the rest of the build uses
(hive.exec.classifier): it carries Hive's own
+ classes without Hive's shaded copies of third-party libraries, so it
does not drag protobuf, guava and
+ friends into this bundle. Scope follows utilities.bundle.hive.scope like
the hive artifacts above, so it
+ is provided by default and only packaged under
-Putilities-bundle-shade-hive.
+ -->
+ <dependency>
+ <groupId>${hive.groupid}</groupId>
+ <artifactId>hive-exec</artifactId>
+ <version>${hive.version}</version>
+ <classifier>${hive.exec.classifier}</classifier>
+ <scope>${utilities.bundle.hive.scope}</scope>
+ <!--
+ Deliberately no <exclusions> here. The root pom's dependencyManagement
entry for hive-exec already
+ excludes javax.mail, the jetty aggregate, pentaho, log4j 1.x, log4j2,
slf4j-log4j12 and hbase, several
+ of which the enforcer bans outright. Declaring exclusions locally
replaces that managed set rather
+ than adding to it, which would quietly let those back in.
+ -->
Review Comment:
Confirmed, and reverted: the pom change is gone entirely, so the default
bundle and its commons-io 2.14.0 are untouched. Thanks for catching that,
shipping a CVE regression in the jar CI builds would have been much worse than
the bug being fixed.
##########
packaging/hudi-utilities-bundle/pom.xml:
##########
@@ -158,6 +158,8 @@
<include>org.apache.hive:hive-service-rpc</include>
<include>org.apache.hive:hive-metastore</include>
<include>org.apache.hive:hive-jdbc</include>
+ <!-- MapredParquetInputFormat, superclass of the bundled
hudi-hadoop-mr input formats -->
Review Comment:
Fixed at the call site instead, so the slim and spark bundles are covered by
the same change and "Closes #16374" now holds for all three.
##########
packaging/hudi-utilities-bundle/pom.xml:
##########
@@ -158,6 +158,8 @@
<include>org.apache.hive:hive-service-rpc</include>
<include>org.apache.hive:hive-metastore</include>
<include>org.apache.hive:hive-jdbc</include>
+ <!-- MapredParquetInputFormat, superclass of the bundled
hudi-hadoop-mr input formats -->
+ <include>org.apache.hive:hive-exec</include>
Review Comment:
Moot now: no hive-exec is bundled, so no un-relocated `ql` classes ship and
the `HiveConf` rewrite problem from #961 cannot arise.
##########
packaging/hudi-utilities-bundle/pom.xml:
##########
@@ -158,6 +158,8 @@
<include>org.apache.hive:hive-service-rpc</include>
<include>org.apache.hive:hive-metastore</include>
<include>org.apache.hive:hive-jdbc</include>
+ <!-- MapredParquetInputFormat, superclass of the bundled
hudi-hadoop-mr input formats -->
+ <include>org.apache.hive:hive-exec</include>
Review Comment:
Moot now that no hive-exec ships, so `file://` binding is unchanged.
##########
packaging/hudi-utilities-bundle/pom.xml:
##########
@@ -355,6 +357,31 @@
<scope>${utilities.bundle.hive.scope}</scope>
</dependency>
+ <!--
+ Supplies org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat,
which is the superclass of
+ HoodieParquetInputFormatBase in hudi-hadoop-mr, which is whitelisted in
the shade artifactSet above. Without
+ this, the bundle ships those subclasses with their superclass missing
and any use of the Hive input formats fails
+ with ClassNotFoundException on MapredParquetInputFormat (HUDI-7321).
+
+ The core classifier is what the rest of the build uses
(hive.exec.classifier): it carries Hive's own
+ classes without Hive's shaded copies of third-party libraries, so it
does not drag protobuf, guava and
+ friends into this bundle. Scope follows utilities.bundle.hive.scope like
the hive artifacts above, so it
+ is provided by default and only packaged under
-Putilities-bundle-shade-hive.
Review Comment:
Moot for the profile, which is untouched again. Coverage instead lives in
`TestHoodieInputFormatUtils`, which runs in CI and pins each constant against
the class it names, so the sync-time names cannot rot silently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
