This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new aec0e04f983c [SPARK-60897][SQL] Fix Hive 4 metastore clients failing
with `No suitable driver` when another class loader initialized `DriverManager`
aec0e04f983c is described below
commit aec0e04f983c05746c190921fac4cb66f8b58aad
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed Nov 4 18:57:35 2026 -0800
[SPARK-60897][SQL] Fix Hive 4 metastore clients failing with `No suitable
driver` when another class loader initialized `DriverManager`
### What changes were proposed in this pull request?
This PR makes `HiveClientImpl` load the configured metastore JDBC driver
(`javax.jdo.option.ConnectionDriverName`) through the client class loader when
an isolated client opens the metastore database itself, i.e., when
`hive.metastore.uris` is empty. The driver class registers itself with
`java.sql.DriverManager` in its static initializer.
Since Hive 4, the metastore creates its HikariCP connection pool with only
the JDBC URL, user name, and password (`HikariCPDataSourceProvider`), so
HikariCP gets the driver via `DriverManager.getDriver(url)`. `DriverManager`
- loads the drivers listed in `META-INF/services/java.sql.Driver` only once
per JVM, through the context class loader of its first caller, and
- returns a registered driver only if the caller's class loader resolves
the driver class name to the same class.
So, the Derby driver in the jars of an isolated client
(`spark.sql.hive.metastore.jars` is `maven` or `path`) was registered only by
chance: when `DriverManager` was first used under the client class loader, or
as a side effect of the caller check above, which initializes the client's own
class of the same name as a registered driver.
### Why are the changes needed?
Isolated Hive 4 metastore clients fail with `java.sql.SQLException: No
suitable driver` depending on which class loader used `DriverManager` first in
the JVM.
```
org.apache.hadoop.hive.metastore.api.MetaException: Got exception:
org.apache.hadoop.hive.metastore.api.MetaException java.sql.SQLException: No
suitable driver
...
Caused by: java.lang.RuntimeException: Failed to get driver instance for
jdbcUrl=jdbc:derby:;databaseName=...;create=true
at
org.apache.hive.com.zaxxer.hikari.util.DriverDataSource.<init>(DriverDataSource.java:114)
...
at
org.apache.hadoop.hive.metastore.datasource.HikariCPDataSourceProvider.create(HikariCPDataSourceProvider.java:93)
...
Caused by: java.sql.SQLException: No suitable driver
at java.sql/java.sql.DriverManager.getDriver(DriverManager.java:300)
at
org.apache.hive.com.zaxxer.hikari.util.DriverDataSource.<init>(DriverDataSource.java:106)
```
- A Hive 4.2 client (Derby 10.17.1.0,
`org.apache.derby.iapi.jdbc.AutoloadedDriver`) fails when an older Hive client
(Derby 10.14.2.0, `org.apache.derby.jdbc.AutoloadedDriver`) used
`DriverManager` first. For example, `HiveClientSuite(4.2)` fails 69 of 75 tests
when `HiveClientSuites` runs alone, because the Hive 2.3.10 client of `success
sanity check` initializes `DriverManager`.
- A Hive 4.0 or 4.1 client (Derby 10.14.2.0) fails when only the drivers of
the application class loader, e.g. Spark's own Derby 10.17.1.0, are registered,
i.e., when the application used `DriverManager` before the first metastore
access.
The CI passes `HiveClientSuite(4.2)` only because other suites in the same
JVM register Spark's own Derby 10.17.1.0 driver first, whose class name is the
same as the one of Hive 4.2's Derby. In that case, the first connection attempt
fails, and Hive's internal retry succeeds.
Hive 4 metastore support was added by the following PRs.
- Hive 4.0: #48823 (SPARK-45265), since Apache Spark
[4.0.0](https://github.com/apache/spark/releases/tag/v4.0.0) (2025-05-19)
- Hive 4.1: #51809 (SPARK-53095), since Apache Spark
[4.1.0](https://github.com/apache/spark/releases/tag/v4.1.0) (2025-12-11)
- Hive 4.2: #56337 (SPARK-57263), not released yet
Since the released Hive 4.0 and 4.1 metastore clients are affected, this PR
should be backported to `branch-4.x`, `branch-4.3`, `branch-4.2`, `branch-4.1`,
and `branch-4.0`.
### Does this PR introduce _any_ user-facing change?
Yes. Hive 4.0 and 4.1 metastore clients that Spark loads from
`spark.sql.hive.metastore.jars` (`maven` or `path`) and that open a local
metastore database no longer fail with `No suitable driver` when
`DriverManager` was used before the first metastore access. Hive 4.2 metastore
support is not released yet.
### How was this patch tested?
Pass the CIs with the existing tests, and manually on JDK 21.
```
$ SPARK_TEST_HIVE_CLIENT_VERSIONS=4.2 build/sbt -Phive \
"hive/testOnly *.HiveClientSuites *.HiveClientUserNameSuites
*.HivePartitionFilteringSuites"
```
Before this PR, `HiveClientSuite(4.2)` failed 69 of 75 tests with `No
suitable driver`. After this PR, all 126 tests pass.
```
$ build/sbt -Phive "hive/testOnly *.HiveClientSuites"
```
After this PR, all 678 tests of Hive 2.0 to 4.2 pass.
In addition, with a temporary test change that opens an in-memory Derby
connection through the application class loader before any Hive client is
created, `4.1: createDatabase` failed with `No suitable driver` before this PR
and passes after this PR.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #60113 from dongjoon-hyun/SPARK-60897.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 79f5485b1cb51be1b41fe1d20935f6644085642c)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../spark/sql/hive/client/HiveClientImpl.scala | 21 +++++++++++++++++++++
1 file changed, 21 insertions(+)
diff --git
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
index 8db1a338137b..871f54f973db 100644
---
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
+++
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
@@ -178,6 +178,27 @@ private[hive] class HiveClientImpl(
log"${MDC(HIVE_CLIENT_VERSION, version.fullVersion)}) is " +
log"${MDC(PATH,
conf.getVar(HiveConf.getConfVars("hive.metastore.warehouse.dir")))}")
+ // Hive 4 creates its metastore connection pool without a JDBC driver class
name, so the pool
+ // gets the driver from java.sql.DriverManager. DriverManager loads the
drivers listed in
+ // META-INF/services only once per JVM, through the context class loader of
its first caller,
+ // and returns only drivers that the caller's class loader resolves to the
same class. When an
+ // isolated client opens the metastore database itself, load the configured
JDBC driver through
+ // the client class loader so that the driver registers itself with
DriverManager.
+ if (clientLoader.isolationOn) {
+ val msUri = conf.getVar(HiveConf.getConfVars("hive.metastore.uris"))
+ if (msUri == null || msUri.trim.isEmpty) {
+ val driver =
conf.getVar(HiveConf.getConfVars("javax.jdo.option.ConnectionDriverName"))
+ try {
+ // scalastyle:off classforname
+ Class.forName(driver, true, clientLoader.classLoader)
+ // scalastyle:on classforname
+ } catch {
+ case NonFatal(e) =>
+ logWarning(log"Failed to load the metastore JDBC driver
${MDC(CLASS_NAME, driver)}", e)
+ }
+ }
+ }
+
private def newState(): SessionState = {
val hiveConf = newHiveConf(sparkConf, hadoopConf, extraConfig,
Some(initClassLoader))
val state = new SessionState(hiveConf)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]