voonhous commented on code in PR #19682:
URL: https://github.com/apache/hudi/pull/19682#discussion_r3821167583
##########
pom.xml:
##########
@@ -128,6 +128,9 @@
<hadoop.version>2.10.2</hadoop.version>
<hive.groupid>org.apache.hive</hive.groupid>
<hive.version>2.3.10</hive.version>
+ <!-- The libthrift the Hive client jars above are compiled against (Hive
2.3.10 bumped
+ it for CVE-2020-13949); keep in lockstep with hive.version. See
#19680. -->
+ <hive.libthrift.version>0.14.1</hive.libthrift.version>
Review Comment:
Verified against the published spark-parent poms: 4.0.2, 4.1.1 and 4.2.0 all
pin libthrift 0.16.0 alongside the same Hive 2.3.10 client, so the direct pin
was indeed downgrading Spark 4 builds. Pushed d64e7419f4cd which overrides
hive.libthrift.version to 0.16.0 in the spark4.0/4.1/4.2 profiles and keeps
0.14.1 as the Spark 3 minimum.
##########
hudi-spark-datasource/hudi-spark/pom.xml:
##########
@@ -332,6 +332,31 @@
</dependency>
<!-- Hive -->
+ <!-- Direct pin: the Hive 2.3.10 client jars are compiled against
libthrift 0.14.1
+ (HiveAuthUtils.getSocketTransport uses TConfiguration), but
dependency mediation
+ otherwise picks 0.12.0 from spark-hive, which lacks that class and
turns every
+ hive-jdbc connect into a NoClassDefFoundError. See #19680. -->
+ <dependency>
+ <groupId>org.apache.thrift</groupId>
+ <artifactId>libthrift</artifactId>
+ <version>${hive.libthrift.version}</version>
Review Comment:
Addressed in d64e7419f4cd. The bundle now declares libthrift and libfb303 at
${spark.bundle.hive.scope}, adds both to the shade include list, and relocates
org.apache.thrift. and com.facebook.fb303. under the hive shade prefix. fb303
has to come along because the generated hive-metastore client extends
FacebookService.Client, so relocating thrift alone would have fixed the jdbc
path while breaking metastore-mode sync under the profile. The default build is
unchanged: provided scope keeps both jars out of the bundle and the relocations
are no-ops with the empty prefix.
hudi-utilities-bundle has the same gap under its own shade-hive profile;
I'll handle that separately to keep this one scoped to hudi-spark and its
bundle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
