voonhous opened a new issue, #19680:
URL: https://github.com/apache/hudi/issues/19680
**Describe the problem**
Hive 2.3.10 bumped its libthrift to 0.14.1 (CVE-2020-13949), so the Hive
2.3.10 client jars hudi builds against are compiled against thrift 0.14.1 APIs:
`hive-common`'s `HiveAuthUtils.getSocketTransport` does `new TSocket(new
TConfiguration(), host, port, timeout)`. In hudi-spark, maven dependency
mediation resolves libthrift 0.12.0 instead (nearest win via
`spark-hive_2.12:3.5.5 -> libthrift:0.12.0`; the hive jars' 0.14.1 is "omitted
for conflict with 0.12.0"), and 0.12.0 has no `TConfiguration`.
As a result, any `DriverManager.getConnection("jdbc:hive2://...")` on this
classpath dies in transport setup instead of connecting or failing with an
`SQLException`:
```
java.lang.NoClassDefFoundError: org/apache/thrift/TConfiguration
at
org.apache.hadoop.hive.common.auth.HiveAuthUtils.getSocketTransport(HiveAuthUtils.java:47)
at
org.apache.hive.jdbc.HiveConnection.createUnderlyingTransport(HiveConnection.java:519)
at
org.apache.hive.jdbc.HiveConnection.createBinaryTransport(HiveConnection.java:539)
at
org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:309)
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:196)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677)
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:228)
at ReproHive2.main(ReproHive2.java:6)
Caused by: java.lang.ClassNotFoundException: org.apache.thrift.TConfiguration
at
java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
at
java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
... 9 more
```
(Repro: `DriverManager.getConnection("jdbc:hive2://unused", "", "x")` on
hudi-spark's resolved test classpath from `mvn dependency:build-classpath
-Dmdep.includeScope=test`.)
Concretely, `sync_validate`'s `complete` / `latestPartitions` record-count
modes are unusable on this dependency set, and in `complete` mode the error is
further masked into an NPE by #19635. It is also why the fail-fast pins in
`TestValidateHoodieSyncProcedure` fail deterministically wherever libthrift
0.12.0 wins resolution, see #19679.
**Suggested fix**
Align libthrift with the Hive 2.3.10 client jars, e.g. a
dependencyManagement pin or direct dependency on libthrift 0.14.1 in
hudi-spark. 0.14.1 keeps the `TSocket(String, int, int)` constructor spark-hive
3.5's 0.12.0-era callers use, but the spark-hive / thriftserver and HMS-based
test suites need a verification pass before claiming the bump safe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
