This is an automated email from the ASF dual-hosted git repository.
voonhous pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/master by this push:
new 2c14ff2779dd fix(spark): align libthrift with the Hive 2.3.10 client
jars (#19682)
2c14ff2779dd is described below
commit 2c14ff2779ddd59e58bf67219a965ccc71ad4b7a
Author: voonhous <[email protected]>
AuthorDate: Fri Aug 21 15:43:42 2026 +0800
fix(spark): align libthrift with the Hive 2.3.10 client jars (#19682)
* fix(spark): align libthrift with the Hive 2.3.10 client jars
Hive 2.3.10 is compiled against libthrift 0.14.1
(HiveAuthUtils.getSocketTransport uses TConfiguration), but in hudi-spark
dependency mediation picked 0.12.0 from spark-hive, so every hive-jdbc connect
died with NoClassDefFoundError: TConfiguration instead of connecting or failing
with an SQLException. Pin libthrift 0.14.1 directly in hudi-spark, with
exclusions so the only classpath change is the libthrift version itself.
Closes #19680
* review(spark): keep the Spark 4 libthrift and bundle thrift under
shade-hive
The spark4.0/4.1/4.2 profiles now override hive.libthrift.version to
0.16.0, the
version those Spark releases pair with the same Hive 2.3.10 client, so the
direct
pin no longer downgrades the Spark-selected thrift. 0.14.1 stays the Spark 3
minimum.
Under -Pspark-bundle-shade-hive the bundle now embeds and relocates
libthrift and
libfb303 next to the shaded Hive client jars, so the relocated classes stop
binding to Spark's provided libthrift 0.12.0 and hitting the same missing
TConfiguration. fb303 rides along because the generated hive-metastore
client
extends its classes; relocating thrift alone would break metastore-mode
sync.
The default build is unchanged: the new deps are provided scope and the
relocations are no-ops with the empty shade prefix.
---
hudi-spark-datasource/hudi-spark/pom.xml | 25 ++++++++++++++++++++++++
packaging/hudi-spark-bundle/pom.xml | 33 ++++++++++++++++++++++++++++++++
pom.xml | 16 ++++++++++++++++
3 files changed, 74 insertions(+)
diff --git a/hudi-spark-datasource/hudi-spark/pom.xml
b/hudi-spark-datasource/hudi-spark/pom.xml
index bf7e2dbc5ce3..8b3daf01fdfe 100644
--- a/hudi-spark-datasource/hudi-spark/pom.xml
+++ b/hudi-spark-datasource/hudi-spark/pom.xml
@@ -332,6 +332,31 @@
</dependency>
<!-- Hive -->
+ <!-- Direct pin: the Hive 2.3.10 client jars are compiled against
libthrift 0.14.1
+ (HiveAuthUtils.getSocketTransport uses TConfiguration), but
dependency mediation
+ otherwise picks 0.12.0 from spark-hive, which lacks that class and
turns every
+ hive-jdbc connect into a NoClassDefFoundError. See #19680. -->
+ <dependency>
+ <groupId>org.apache.thrift</groupId>
+ <artifactId>libthrift</artifactId>
+ <version>${hive.libthrift.version}</version>
+ <exclusions>
+ <!-- Servlet-side extras of libthrift; excluded so the only classpath
change
+ against the mediated 0.12.0 is the libthrift version itself. -->
+ <exclusion>
+ <groupId>org.apache.tomcat.embed</groupId>
+ <artifactId>tomcat-embed-core</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.tomcat</groupId>
+ <artifactId>tomcat-annotations-api</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>javax.annotation</groupId>
+ <artifactId>javax.annotation-api</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
<dependency>
<groupId>${hive.groupid}</groupId>
<artifactId>hive-exec</artifactId>
diff --git a/packaging/hudi-spark-bundle/pom.xml
b/packaging/hudi-spark-bundle/pom.xml
index c33ca70b0302..9acec589c65e 100644
--- a/packaging/hudi-spark-bundle/pom.xml
+++ b/packaging/hudi-spark-bundle/pom.xml
@@ -116,6 +116,11 @@
<include>org.apache.hive:hive-service-rpc</include>
<include>org.apache.hive:hive-metastore</include>
<include>org.apache.hive:hive-jdbc</include>
+ <!-- Thrift runtime matching the Hive client jars above;
only bundled under
+ spark-bundle-shade-hive (compile scope), where the
relocated Hive classes
+ must not bind to the older Spark-provided libthrift.
See #19680. -->
+ <include>org.apache.thrift:libthrift</include>
+ <include>org.apache.thrift:libfb303</include>
<include>org.apache.curator:curator-framework</include>
<include>org.apache.curator:curator-client</include>
@@ -174,6 +179,17 @@
<pattern>org.apache.hadoop.hive.service.</pattern>
<shadedPattern>${spark.bundle.hive.shade.prefix}org.apache.hadoop.hive.service.</shadedPattern>
</relocation>
+ <!-- Keep the relocated Hive classes on their matching thrift
runtime instead of
+ the older Spark-provided libthrift; fb303 rides along
because the generated
+ hive-metastore client extends its classes. No-op when the
prefix is empty. -->
+ <relocation>
+ <pattern>org.apache.thrift.</pattern>
+
<shadedPattern>${spark.bundle.hive.shade.prefix}org.apache.thrift.</shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>com.facebook.fb303.</pattern>
+
<shadedPattern>${spark.bundle.hive.shade.prefix}com.facebook.fb303.</shadedPattern>
+ </relocation>
<relocation>
<pattern>com.codahale.metrics.</pattern>
<shadedPattern>org.apache.hudi.com.codahale.metrics.</shadedPattern>
@@ -304,6 +320,23 @@
<scope>${spark.bundle.hive.scope}</scope>
</dependency>
+ <!-- Thrift runtime matching the Hive client jars above. Declared directly
(rather than
+ relying on the transitive versions) so its scope tracks the Hive
jars: provided by
+ default, bundled and relocated under spark-bundle-shade-hive. See
#19680. -->
+ <dependency>
+ <groupId>org.apache.thrift</groupId>
+ <artifactId>libthrift</artifactId>
+ <version>${hive.libthrift.version}</version>
+ <scope>${spark.bundle.hive.scope}</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.thrift</groupId>
+ <artifactId>libfb303</artifactId>
+ <version>${hive.libfb303.version}</version>
+ <scope>${spark.bundle.hive.scope}</scope>
+ </dependency>
+
<!-- zookeeper -->
<dependency>
<groupId>org.apache.curator</groupId>
diff --git a/pom.xml b/pom.xml
index ec7c9c7cafe9..64072dbd20af 100644
--- a/pom.xml
+++ b/pom.xml
@@ -128,6 +128,13 @@
<hadoop.version>2.10.2</hadoop.version>
<hive.groupid>org.apache.hive</hive.groupid>
<hive.version>2.3.10</hive.version>
+ <!-- The thrift artifacts the Hive client jars above are compiled against
(Hive 2.3.10
+ bumped libthrift for CVE-2020-13949); keep in lockstep with
hive.version. The spark4
+ profiles override hive.libthrift.version to the libthrift Spark 4.x
itself ships with
+ the same Hive 2.3.10 client, so the pin never downgrades the
Spark-selected version.
+ See #19680. -->
+ <hive.libthrift.version>0.14.1</hive.libthrift.version>
+ <hive.libfb303.version>0.9.3</hive.libfb303.version>
<hive.parquet.version>1.10.1</hive.parquet.version>
<hive.avro.version>1.11.4</hive.avro.version>
<presto.version>0.273</presto.version>
@@ -2886,6 +2893,9 @@
<hadoop.version>3.4.0</hadoop.version>
<kafka.version>3.8.0</kafka.version>
<hive.storage.version>2.8.1</hive.storage.version>
+ <!-- Spark 4.x ships libthrift 0.16.0 alongside the same Hive 2.3.10
client;
+ keep the Spark-selected version instead of the Spark 3 minimum.
See #19680. -->
+ <hive.libthrift.version>0.16.0</hive.libthrift.version>
<!-- Lance: Use Spark 4.0-specific artifact (Scala 2.13 only) -->
<lance.spark.artifact>lance-spark-4.0_2.13</lance.spark.artifact>
<lance.skip.tests>false</lance.skip.tests>
@@ -2953,6 +2963,9 @@
<hadoop.version>3.4.2</hadoop.version>
<kafka.version>3.9.1</kafka.version>
<hive.storage.version>2.8.1</hive.storage.version>
+ <!-- Spark 4.x ships libthrift 0.16.0 alongside the same Hive 2.3.10
client;
+ keep the Spark-selected version instead of the Spark 3 minimum.
See #19680. -->
+ <hive.libthrift.version>0.16.0</hive.libthrift.version>
<lance.spark.artifact>lance-spark-4.1_2.13</lance.spark.artifact>
<lance.skip.tests>false</lance.skip.tests>
<!-- Vortex: Use Spark 4.1 artifact (Scala 2.13 only) -->
@@ -3020,6 +3033,9 @@
<hadoop.version>3.5.0</hadoop.version>
<kafka.version>3.9.2</kafka.version>
<hive.storage.version>2.8.1</hive.storage.version>
+ <!-- Spark 4.x ships libthrift 0.16.0 alongside the same Hive 2.3.10
client;
+ keep the Spark-selected version instead of the Spark 3 minimum.
See #19680. -->
+ <hive.libthrift.version>0.16.0</hive.libthrift.version>
<!-- TODO: Enable lance tests on Spark 4.2 -->
<lance.spark.artifact>lance-spark-4.0_2.13</lance.spark.artifact>
<lance.skip.tests>true</lance.skip.tests>
