dongjoon-hyun opened a new pull request, #60113:
URL: https://github.com/apache/spark/pull/60113
### What changes were proposed in this pull request?
This PR makes `HiveClientImpl` load the configured metastore JDBC driver
(`javax.jdo.option.ConnectionDriverName`) through the client class loader when
an isolated client opens the metastore database itself, i.e., when
`hive.metastore.uris` is empty. The driver class registers itself with
`java.sql.DriverManager` in its static initializer.
Since Hive 4, the metastore creates its HikariCP connection pool with only
the JDBC URL, user name, and password (`HikariCPDataSourceProvider`), so
HikariCP gets the driver via `DriverManager.getDriver(url)`. `DriverManager`
- loads the drivers listed in `META-INF/services/java.sql.Driver` only once
per JVM, through the context class loader of its first caller, and
- returns a registered driver only if the caller's class loader resolves the
driver class name to the same class.
So, the Derby driver in the jars of an isolated client
(`spark.sql.hive.metastore.jars` is `maven` or `path`) was registered only by
chance: when `DriverManager` was first used under the client class loader, or
as a side effect of the caller check above, which initializes the client's own
class of the same name as a registered driver.
### Why are the changes needed?
Isolated Hive 4 metastore clients fail with `java.sql.SQLException: No
suitable driver` depending on which class loader used `DriverManager` first in
the JVM.
```
org.apache.hadoop.hive.metastore.api.MetaException: Got exception:
org.apache.hadoop.hive.metastore.api.MetaException java.sql.SQLException: No
suitable driver
...
Caused by: java.lang.RuntimeException: Failed to get driver instance for
jdbcUrl=jdbc:derby:;databaseName=...;create=true
at
org.apache.hive.com.zaxxer.hikari.util.DriverDataSource.<init>(DriverDataSource.java:114)
...
at
org.apache.hadoop.hive.metastore.datasource.HikariCPDataSourceProvider.create(HikariCPDataSourceProvider.java:93)
...
Caused by: java.sql.SQLException: No suitable driver
at java.sql/java.sql.DriverManager.getDriver(DriverManager.java:300)
at
org.apache.hive.com.zaxxer.hikari.util.DriverDataSource.<init>(DriverDataSource.java:106)
```
- A Hive 4.2 client (Derby 10.17.1.0,
`org.apache.derby.iapi.jdbc.AutoloadedDriver`) fails when an older Hive client
(Derby 10.14.2.0, `org.apache.derby.jdbc.AutoloadedDriver`) used
`DriverManager` first. For example, `HiveClientSuite(4.2)` fails 69 of 75 tests
when `HiveClientSuites` runs alone, because the Hive 2.3.10 client of `success
sanity check` initializes `DriverManager`.
- A Hive 4.0 or 4.1 client (Derby 10.14.2.0) fails when only the drivers of
the application class loader, e.g. Spark's own Derby 10.17.1.0, are registered,
i.e., when the application used `DriverManager` before the first metastore
access.
The CI passes `HiveClientSuite(4.2)` only because other suites in the same
JVM register Spark's own Derby 10.17.1.0 driver first, whose class name is the
same as the one of Hive 4.2's Derby. In that case, the first connection attempt
fails, and Hive's internal retry succeeds.
### Does this PR introduce _any_ user-facing change?
Yes. Hive 4.0 and 4.1 metastore clients that Spark loads from
`spark.sql.hive.metastore.jars` (`maven` or `path`) and that open a local
metastore database no longer fail with `No suitable driver` when
`DriverManager` was used before the first metastore access. Hive 4.2 metastore
support is not released yet.
### How was this patch tested?
Pass the CIs with the existing tests, and manually on JDK 21.
```
$ SPARK_TEST_HIVE_CLIENT_VERSIONS=4.2 build/sbt -Phive \
"hive/testOnly *.HiveClientSuites *.HiveClientUserNameSuites
*.HivePartitionFilteringSuites"
```
Before this PR, `HiveClientSuite(4.2)` failed 69 of 75 tests with `No
suitable driver`. After this PR, all 126 tests pass.
```
$ build/sbt -Phive "hive/testOnly *.HiveClientSuites"
```
After this PR, all 678 tests of Hive 2.0 to 4.2 pass.
In addition, with a temporary test change that opens an in-memory Derby
connection through the application class loader before any Hive client is
created, `4.1: createDatabase` failed with `No suitable driver` before this PR
and passes after this PR.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
