This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new aec0e04f983c [SPARK-60897][SQL] Fix Hive 4 metastore clients failing 
with `No suitable driver` when another class loader initialized `DriverManager`
aec0e04f983c is described below

commit aec0e04f983c05746c190921fac4cb66f8b58aad
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed Nov 4 18:57:35 2026 -0800

    [SPARK-60897][SQL] Fix Hive 4 metastore clients failing with `No suitable 
driver` when another class loader initialized `DriverManager`
    
    ### What changes were proposed in this pull request?
    
    This PR makes `HiveClientImpl` load the configured metastore JDBC driver 
(`javax.jdo.option.ConnectionDriverName`) through the client class loader when 
an isolated client opens the metastore database itself, i.e., when 
`hive.metastore.uris` is empty. The driver class registers itself with 
`java.sql.DriverManager` in its static initializer.
    
    Since Hive 4, the metastore creates its HikariCP connection pool with only 
the JDBC URL, user name, and password (`HikariCPDataSourceProvider`), so 
HikariCP gets the driver via `DriverManager.getDriver(url)`. `DriverManager`
    - loads the drivers listed in `META-INF/services/java.sql.Driver` only once 
per JVM, through the context class loader of its first caller, and
    - returns a registered driver only if the caller's class loader resolves 
the driver class name to the same class.
    
    So, the Derby driver in the jars of an isolated client 
(`spark.sql.hive.metastore.jars` is `maven` or `path`) was registered only by 
chance: when `DriverManager` was first used under the client class loader, or 
as a side effect of the caller check above, which initializes the client's own 
class of the same name as a registered driver.
    
    ### Why are the changes needed?
    
    Isolated Hive 4 metastore clients fail with `java.sql.SQLException: No 
suitable driver` depending on which class loader used `DriverManager` first in 
the JVM.
    
    ```
    org.apache.hadoop.hive.metastore.api.MetaException: Got exception: 
org.apache.hadoop.hive.metastore.api.MetaException java.sql.SQLException: No 
suitable driver
    ...
    Caused by: java.lang.RuntimeException: Failed to get driver instance for 
jdbcUrl=jdbc:derby:;databaseName=...;create=true
            at 
org.apache.hive.com.zaxxer.hikari.util.DriverDataSource.<init>(DriverDataSource.java:114)
            ...
            at 
org.apache.hadoop.hive.metastore.datasource.HikariCPDataSourceProvider.create(HikariCPDataSourceProvider.java:93)
    ...
    Caused by: java.sql.SQLException: No suitable driver
            at java.sql/java.sql.DriverManager.getDriver(DriverManager.java:300)
            at 
org.apache.hive.com.zaxxer.hikari.util.DriverDataSource.<init>(DriverDataSource.java:106)
    ```
    
    - A Hive 4.2 client (Derby 10.17.1.0, 
`org.apache.derby.iapi.jdbc.AutoloadedDriver`) fails when an older Hive client 
(Derby 10.14.2.0, `org.apache.derby.jdbc.AutoloadedDriver`) used 
`DriverManager` first. For example, `HiveClientSuite(4.2)` fails 69 of 75 tests 
when `HiveClientSuites` runs alone, because the Hive 2.3.10 client of `success 
sanity check` initializes `DriverManager`.
    - A Hive 4.0 or 4.1 client (Derby 10.14.2.0) fails when only the drivers of 
the application class loader, e.g. Spark's own Derby 10.17.1.0, are registered, 
i.e., when the application used `DriverManager` before the first metastore 
access.
    
    The CI passes `HiveClientSuite(4.2)` only because other suites in the same 
JVM register Spark's own Derby 10.17.1.0 driver first, whose class name is the 
same as the one of Hive 4.2's Derby. In that case, the first connection attempt 
fails, and Hive's internal retry succeeds.
    
    Hive 4 metastore support was added by the following PRs.
    - Hive 4.0: #48823 (SPARK-45265), since Apache Spark 
[4.0.0](https://github.com/apache/spark/releases/tag/v4.0.0) (2025-05-19)
    - Hive 4.1: #51809 (SPARK-53095), since Apache Spark 
[4.1.0](https://github.com/apache/spark/releases/tag/v4.1.0) (2025-12-11)
    - Hive 4.2: #56337 (SPARK-57263), not released yet
    
    Since the released Hive 4.0 and 4.1 metastore clients are affected, this PR 
should be backported to `branch-4.x`, `branch-4.3`, `branch-4.2`, `branch-4.1`, 
and `branch-4.0`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. Hive 4.0 and 4.1 metastore clients that Spark loads from 
`spark.sql.hive.metastore.jars` (`maven` or `path`) and that open a local 
metastore database no longer fail with `No suitable driver` when 
`DriverManager` was used before the first metastore access. Hive 4.2 metastore 
support is not released yet.
    
    ### How was this patch tested?
    
    Pass the CIs with the existing tests, and manually on JDK 21.
    
    ```
    $ SPARK_TEST_HIVE_CLIENT_VERSIONS=4.2 build/sbt -Phive \
      "hive/testOnly *.HiveClientSuites *.HiveClientUserNameSuites 
*.HivePartitionFilteringSuites"
    ```
    
    Before this PR, `HiveClientSuite(4.2)` failed 69 of 75 tests with `No 
suitable driver`. After this PR, all 126 tests pass.
    
    ```
    $ build/sbt -Phive "hive/testOnly *.HiveClientSuites"
    ```
    
    After this PR, all 678 tests of Hive 2.0 to 4.2 pass.
    
    In addition, with a temporary test change that opens an in-memory Derby 
connection through the application class loader before any Hive client is 
created, `4.1: createDatabase` failed with `No suitable driver` before this PR 
and passes after this PR.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #60113 from dongjoon-hyun/SPARK-60897.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
    (cherry picked from commit 79f5485b1cb51be1b41fe1d20935f6644085642c)
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../spark/sql/hive/client/HiveClientImpl.scala      | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git 
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala 
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
index 8db1a338137b..871f54f973db 100644
--- 
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
+++ 
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
@@ -178,6 +178,27 @@ private[hive] class HiveClientImpl(
       log"${MDC(HIVE_CLIENT_VERSION, version.fullVersion)}) is " +
     log"${MDC(PATH, 
conf.getVar(HiveConf.getConfVars("hive.metastore.warehouse.dir")))}")
 
+  // Hive 4 creates its metastore connection pool without a JDBC driver class 
name, so the pool
+  // gets the driver from java.sql.DriverManager. DriverManager loads the 
drivers listed in
+  // META-INF/services only once per JVM, through the context class loader of 
its first caller,
+  // and returns only drivers that the caller's class loader resolves to the 
same class. When an
+  // isolated client opens the metastore database itself, load the configured 
JDBC driver through
+  // the client class loader so that the driver registers itself with 
DriverManager.
+  if (clientLoader.isolationOn) {
+    val msUri = conf.getVar(HiveConf.getConfVars("hive.metastore.uris"))
+    if (msUri == null || msUri.trim.isEmpty) {
+      val driver = 
conf.getVar(HiveConf.getConfVars("javax.jdo.option.ConnectionDriverName"))
+      try {
+        // scalastyle:off classforname
+        Class.forName(driver, true, clientLoader.classLoader)
+        // scalastyle:on classforname
+      } catch {
+        case NonFatal(e) =>
+          logWarning(log"Failed to load the metastore JDBC driver 
${MDC(CLASS_NAME, driver)}", e)
+      }
+    }
+  }
+
   private def newState(): SessionState = {
     val hiveConf = newHiveConf(sparkConf, hadoopConf, extraConfig, 
Some(initClassLoader))
     val state = new SessionState(hiveConf)


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to