This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 2b12ddbf0bf8 [SPARK-60895][SQL] Remove the Java 21 guard for the Hive
4.2 metastore
2b12ddbf0bf8 is described below
commit 2b12ddbf0bf86cdc047dcb228681ee5eacdcceaa
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed Nov 4 19:48:51 2026 -0800
[SPARK-60895][SQL] Remove the Java 21 guard for the Hive 4.2 metastore
### What changes were proposed in this pull request?
This PR removes the "Hive 4.2 requires Java 21 or later" guard that
SPARK-57263 added together with the Hive 4.2 metastore support, and everything
that existed only for it:
- `IsolatedClientLoader`: remove the constructor check that threw
`unsupportedHiveMetastoreVersionForJavaError` when `version == hive.v4_2` and
`!Utils.isJavaVersionAtLeast21`.
- `QueryExecutionErrors`: remove
`unsupportedHiveMetastoreVersionForJavaError`.
- `error-conditions.json`: remove the
`UNSUPPORTED_HIVE_METASTORE_VERSION_FOR_JAVA` error condition.
- `HiveUtils` and `docs/sql-data-sources-hive-tables.md`: remove "Note:
Hive 4.2 requires Java 21 or later." from the description of
`spark.sql.hive.metastore.version`.
- `HiveClientVersions` (test): remove the filter that dropped `4.2` from
`versions` on Java < 21. `versions` is now the
`SPARK_TEST_HIVE_CLIENT_VERSIONS` list, or the default list that includes `4.2`.
### Why are the changes needed?
SPARK-60669 dropped Java 17 support and made Java 21 the minimum, so
`Utils.isJavaVersionAtLeast21` is always true and the guard can never fire.
Spark is compiled with `--release 21` (class file major version 65), so a JVM
older than Java 21 cannot even load `IsolatedClientLoader`. What is left is a
dead branch, an error condition that nothing can raise, and a documentation
note that now applies to all of Spark rather than to Hive 4.2 only.
### Does this PR introduce _any_ user-facing change?
No. `UNSUPPORTED_HIVE_METASTORE_VERSION_FOR_JAVA` was added by SPARK-57263
on `master` and has not been released: it does not exist in `branch-4.x`,
`branch-4.3`, or the `v4.3.0` tag (tagged commit dated 2026-10-27). The removed
guard could only fire on a JVM older than Java 21, which `master` no longer
runs on.
### How was this patch tested?
On JDK 21:
```
build/sbt -Phive catalyst/Test/compile hive/Test/compile
build/sbt -Phive 'core/testOnly org.apache.spark.SparkThrowableSuite'
build/sbt -Phive 'hive/testOnly org.apache.spark.sql.hive.HiveUtilsSuite'
```
Also checked that nothing references the removed code, and that the error
condition is not in any release:
```
git grep -n -e UNSUPPORTED_HIVE_METASTORE_VERSION_FOR_JAVA -e
unsupportedHiveMetastoreVersionForJava
git grep -n isJavaVersionAtLeast21 -- sql/hive
git grep -n UNSUPPORTED_HIVE_METASTORE_VERSION_FOR_JAVA v4.3.0 --
common/utils/src/main/resources/error/error-conditions.json
```
All three print nothing. The same check on `branch-4.x` and `branch-4.3`
also prints nothing.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Sonnet 5.5
Closes #60110 from dongjoon-hyun/SPARK-60895.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
common/utils/src/main/resources/error/error-conditions.json | 6 ------
docs/sql-data-sources-hive-tables.md | 1 -
.../org/apache/spark/sql/errors/QueryExecutionErrors.scala | 12 ------------
.../src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala | 3 +--
.../apache/spark/sql/hive/client/IsolatedClientLoader.scala | 8 --------
.../apache/spark/sql/hive/client/HiveClientVersions.scala | 12 +-----------
6 files changed, 2 insertions(+), 40 deletions(-)
diff --git a/common/utils/src/main/resources/error/error-conditions.json
b/common/utils/src/main/resources/error/error-conditions.json
index b25edfd3a790..0cb7bc927152 100644
--- a/common/utils/src/main/resources/error/error-conditions.json
+++ b/common/utils/src/main/resources/error/error-conditions.json
@@ -9812,12 +9812,6 @@
},
"sqlState" : "0A000"
},
- "UNSUPPORTED_HIVE_METASTORE_VERSION_FOR_JAVA" : {
- "message" : [
- "Hive metastore version <version> requires Java <requiredJavaVersion> or
later, but the current JVM is Java <currentJavaVersion>. Please upgrade your
Java version or use an earlier Hive metastore version."
- ],
- "sqlState" : "0A000"
- },
"UNSUPPORTED_HIVE_TYPE" : {
"message" : [
"Cannot read the Hive type <fieldType> of the column <fieldName> because
Spark SQL does not support this data type."
diff --git a/docs/sql-data-sources-hive-tables.md
b/docs/sql-data-sources-hive-tables.md
index f4587ee41eaa..a1886ea2affb 100644
--- a/docs/sql-data-sources-hive-tables.md
+++ b/docs/sql-data-sources-hive-tables.md
@@ -131,7 +131,6 @@ The following options can be used to configure the version
of Hive that is used
<td>
Version of the Hive metastore. Available
options are <code>2.0.0</code> through <code>2.3.10</code>,
<code>3.0.0</code> through <code>3.1.3</code>, and <code>4.0.0</code> through
<code>4.2.0</code>.
- Note: Hive 4.2 requires Java 21 or later.
</td>
<td>1.4.0</td>
</tr>
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
index 381bbc634cdf..7e06a764fc0b 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
@@ -1928,18 +1928,6 @@ private[sql] object QueryExecutionErrors extends
QueryErrorsBase with ExecutionE
"key" -> key))
}
- def unsupportedHiveMetastoreVersionForJavaError(
- version: String,
- requiredJavaVersion: Int,
- currentJavaVersion: Int): SparkUnsupportedOperationException = {
- new SparkUnsupportedOperationException(
- errorClass = "UNSUPPORTED_HIVE_METASTORE_VERSION_FOR_JAVA",
- messageParameters = Map(
- "version" -> version,
- "requiredJavaVersion" -> requiredJavaVersion.toString,
- "currentJavaVersion" -> currentJavaVersion.toString))
- }
-
def loadHiveClientCausesNoClassDefFoundError(
cnf: NoClassDefFoundError,
execJars: Seq[URL],
diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
index d5dceaa8be32..9e2c0f5f42f0 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
@@ -77,8 +77,7 @@ private[spark] object HiveUtils extends Logging {
.doc("Version of the Hive metastore. Available options are " +
"<code>2.0.0</code> through <code>2.3.10</code>, " +
"<code>3.0.0</code> through <code>3.1.3</code> and " +
- "<code>4.0.0</code> through <code>4.2.0</code>. " +
- "Note: Hive 4.2 requires Java 21 or later.")
+ "<code>4.0.0</code> through <code>4.2.0</code>.")
.version("1.4.0")
.stringConf
.checkValue(isCompatibleHiveVersion, "Unsupported Hive Metastore version")
diff --git
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
index 86c831a50949..bfc302028d5e 100644
---
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
+++
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
@@ -220,14 +220,6 @@ private[hive] class IsolatedClientLoader(
val barrierPrefixes: Seq[String] = Seq.empty)
extends Logging {
- // Hive 4.2 requires Java 21 or later. The guard lives on the
client-construction path rather
- // than in IsolatedClientLoader.hiveVersion, which is also used for
version-string validation,
- // so the actionable message reaches the user instead of being swallowed.
- if (version == hive.v4_2 && !Utils.isJavaVersionAtLeast21) {
- throw QueryExecutionErrors.unsupportedHiveMetastoreVersionForJavaError(
- version.fullVersion, 21, Runtime.version().feature())
- }
-
/**
* This controls whether the generated clients maintain an
independent/isolated copy of the
* Hive `SessionState`. If false, the Hive will leverage the global/static
copy of
diff --git
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientVersions.scala
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientVersions.scala
index 0ea4834839e5..15b0a09f0bb3 100644
---
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientVersions.scala
+++
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientVersions.scala
@@ -17,21 +17,11 @@
package org.apache.spark.sql.hive.client
-import org.apache.spark.util.Utils
-
private[client] trait HiveClientVersions {
private val testVersions = sys.env.get("SPARK_TEST_HIVE_CLIENT_VERSIONS")
- private val allVersions = if (testVersions.nonEmpty) {
+ protected val versions: IndexedSeq[String] = if (testVersions.nonEmpty) {
testVersions.get.split(",").map(_.trim).filter(_.nonEmpty).toIndexedSeq
} else {
IndexedSeq("2.0", "2.1", "2.2", "2.3", "3.0", "3.1", "4.0", "4.1", "4.2")
}
-
- protected val versions: IndexedSeq[String] = {
- if (Utils.isJavaVersionAtLeast21) {
- allVersions
- } else {
- allVersions.filterNot(_ == "4.2")
- }
- }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]