This is an automated email from the ASF dual-hosted git repository.
yuqi1129 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new 768213d305 [#11353] fix(hive): fix three ClassLoader/Kerberos bugs
causing HMS connection failure (#11355)
768213d305 is described below
commit 768213d3057540319856c83c0c646b6e690ebe91
Author: weijiajun <[email protected]>
AuthorDate: Thu Jun 4 21:06:36 2026 +0800
[#11353] fix(hive): fix three ClassLoader/Kerberos bugs causing HMS
connection failure (#11355)
### What changes were proposed in this pull request?
This PR fixes three independent bugs in `HiveClientFactory` that
together prevent Hive Metastore
connections from working when Kerberos authentication is enabled and
impersonation is **disabled**.
**Changes:**
1. **`createHiveClientWithBackend()`** – Use
`HiveClientFactory.class.getClassLoader()` as
`baseLoader` instead of `Thread.currentThread().getContextClassLoader()`
(TCCL). TCCL is
unstable across threads; using it causes `UserGroupInformation` to be
loaded by two different
ClassLoaders, making the TGT stored after `login()` invisible at
connection time.
2. **`createHiveClientImpl()`** – Load `HiveVersion` from the isolated
`HiveClientClassLoader`
before calling `getConstructor()`. `HiveClientImpl` is a barrier class
redefined via
`defineClass` inside the isolated ClassLoader; its constructor expects
the `HiveVersion` type
from the isolated ClassLoader scope. Passing the system ClassLoader's
`HiveVersion.class`
causes `NoSuchMethodException`.
3. **`createHiveClientInternal()`** – Add a `realUgi.doAs()` wrapper for
the non-impersonation
Kerberos branch. The JVM default
`javax.security.auth.useSubjectCredsOnly=true` means GSSAPI
only finds credentials in the current thread's JAAS Subject context.
`KerberosClient.login()`
stores the TGT in `realLoginUgi.subject` but does not bind it to the
current thread;
`ugi.doAs()` is required to do so. The impersonation branch already does
this correctly via
`createProxyHiveClientImpl`; this fix brings the non-impersonation path
in line.
Also adds `KerberosClient.getRealLoginUgi()` as the public accessor
needed by fix #3, and adds
`"Cannot find Hive jar directory"` to the Hive2 fallback condition to
handle environments where
HIVE3 libs directory is absent.
### Why are the changes needed?
Without these fixes, any Hive catalog configured with Kerberos
authentication and impersonation
disabled fails to connect to HMS with one or more of:
```
GSS initiate failed
No valid credentials provided (Mechanism level: Failed to find any Kerberos
tgt)
```
```
java.lang.NoSuchMethodException: HiveClientImpl.<init>(HiveVersion,
Properties)
```
All three bugs exist on the same code path and must be fixed together
for the feature to work.
Fix: #11353
### Does this PR introduce _any_ user-facing change?
No API or configuration changes. This is a bug fix for an existing
feature
(Kerberos HMS authentication without impersonation) that was silently
broken.
### How was this patch tested?
Verified on an internal deployment with a real Kerberos-secured HMS
(non-impersonation mode):
- `GET /api/metalakes/{metalake}/catalogs/{catalog}/schemas` returns
schema list correctly after fix
- `GET
/api/metalakes/{metalake}/catalogs/{catalog}/schemas/{schema}/tables`
returns table list correctly after fix
- No `GSSException` or `NoSuchMethodException` in server logs
Unit tests in `hive-metastore-common` all pass (`./gradlew
:catalogs:hive-metastore-common:test -PskipITs`).
Note: a full Kerberos integration test requires a KDC environment which
is not available in
standard CI. A follow-up issue can track adding a Docker-based KDC
integration test.
---
catalogs/hive-metastore-common/build.gradle.kts | 1 +
.../gravitino/hive/client/HiveClientFactory.java | 27 ++++++++--
.../gravitino/hive/kerberos/KerberosClient.java | 13 +++++
.../TestHive2HMSWithKerberosNoImpersonation.java | 59 ++++++++++++++++++++++
4 files changed, 95 insertions(+), 5 deletions(-)
diff --git a/catalogs/hive-metastore-common/build.gradle.kts
b/catalogs/hive-metastore-common/build.gradle.kts
index 9e2c387363..1479f18e96 100644
--- a/catalogs/hive-metastore-common/build.gradle.kts
+++ b/catalogs/hive-metastore-common/build.gradle.kts
@@ -132,6 +132,7 @@ dependencies {
testImplementation(libs.caffeine)
testImplementation(libs.junit.jupiter.api)
testImplementation(libs.mockito.core)
+ testImplementation(libs.mockito.inline)
testImplementation(libs.testcontainers)
testImplementation(libs.woodstox.core)
diff --git
a/catalogs/hive-metastore-common/src/main/java/org/apache/gravitino/hive/client/HiveClientFactory.java
b/catalogs/hive-metastore-common/src/main/java/org/apache/gravitino/hive/client/HiveClientFactory.java
index c80c2590ce..3a2a80898d 100644
---
a/catalogs/hive-metastore-common/src/main/java/org/apache/gravitino/hive/client/HiveClientFactory.java
+++
b/catalogs/hive-metastore-common/src/main/java/org/apache/gravitino/hive/client/HiveClientFactory.java
@@ -27,6 +27,7 @@ import static
org.apache.gravitino.hive.client.Util.updateConfigurationFromPrope
import com.google.common.base.Preconditions;
import java.lang.reflect.Constructor;
import java.lang.reflect.Method;
+import java.security.PrivilegedExceptionAction;
import java.util.Properties;
import org.apache.commons.lang3.reflect.MethodUtils;
import org.apache.gravitino.exceptions.GravitinoRuntimeException;
@@ -115,10 +116,13 @@ public final class HiveClientFactory {
public HiveClient createHiveClientWithBackend() {
HiveClient client = null;
HiveClientClassLoader classloader = null;
+ // Use HiveClientFactory's own ClassLoader as baseLoader so that shared
classes
+ // (e.g. HiveClient interface, UserGroupInformation) are resolved
consistently
+ // regardless of which thread calls this method (TCCL is not stable across
threads).
+ ClassLoader factoryCl = HiveClientFactory.class.getClassLoader();
try {
// Try using Hive3 first
- classloader =
- HiveClientClassLoader.createLoader(HIVE3,
Thread.currentThread().getContextClassLoader());
+ classloader = HiveClientClassLoader.createLoader(HIVE3, factoryCl);
client = createHiveClientInternal(classloader);
client.getCatalogs();
LOG.info("Connected to Hive Metastore using Hive version HIVE3");
@@ -137,10 +141,9 @@ public final class HiveClientFactory {
// Fallback to Hive2 if we can list databases
if (e.getMessage().contains("Invalid method name: 'get_catalogs'")
|| e.getMessage().contains("class not found") // caused by
MiniHiveMetastoreService
+ || e.getMessage().contains("Cannot find Hive jar directory") //
HIVE3 libs dir absent
) {
- classloader =
- HiveClientClassLoader.createLoader(
- HIVE2, Thread.currentThread().getContextClassLoader());
+ classloader = HiveClientClassLoader.createLoader(HIVE2, factoryCl);
client = createHiveClientInternal(classloader);
LOG.info("Connected to Hive Metastore using Hive version HIVE2");
backendClassLoader = classloader;
@@ -162,6 +165,9 @@ public final class HiveClientFactory {
HiveClientClassLoader.HiveVersion version, Properties properties,
ClassLoader classloader)
throws Exception {
Class<?> hiveClientImplClass =
classloader.loadClass(HiveClientImpl.class.getName());
+ // HiveVersion is a shared class (isSharedClass covers all
org.apache.gravitino.* classes),
+ // so the isolated classloader delegates to the base classloader and both
sides hold the
+ // same Class object. We can pass HiveVersion.class directly to
getConstructor().
Constructor<?> hiveClientImplCtor =
hiveClientImplClass.getConstructor(
HiveClientClassLoader.HiveVersion.class, Properties.class);
@@ -202,6 +208,17 @@ public final class HiveClientFactory {
return createProxyHiveClientImpl(
classloader.getHiveVersion(), properties, ugi, classloader);
+ } else if (enableKerberos) {
+ // UGI is a shared class (org.apache.hadoop.* delegated to
baseLoader), so the system CL
+ // and HiveClientClassLoader share the same UGI static state. The TGT
is already stored in
+ // realLoginUgi.subject by kerberosClient.login(). The only thing
needed is to bind that
+ // Subject to the current thread so GSSAPI can find the TGT during the
HMS Thrift handshake.
+ // UGI.doAs() wraps Subject.doAs() internally — same pattern as
ImpalaEngineAdapter.
+ UserGroupInformation realUgi = kerberosClient.getRealLoginUgi();
+ final HiveClientClassLoader.HiveVersion hiveVersion =
classloader.getHiveVersion();
+ return realUgi.doAs(
+ (PrivilegedExceptionAction<HiveClient>)
+ () -> createHiveClientImpl(hiveVersion, properties,
classloader));
} else {
return createHiveClientImpl(classloader.getHiveVersion(), properties,
classloader);
}
diff --git
a/catalogs/hive-metastore-common/src/main/java/org/apache/gravitino/hive/kerberos/KerberosClient.java
b/catalogs/hive-metastore-common/src/main/java/org/apache/gravitino/hive/kerberos/KerberosClient.java
index cb115048e4..a86535184a 100644
---
a/catalogs/hive-metastore-common/src/main/java/org/apache/gravitino/hive/kerberos/KerberosClient.java
+++
b/catalogs/hive-metastore-common/src/main/java/org/apache/gravitino/hive/kerberos/KerberosClient.java
@@ -198,4 +198,17 @@ public class KerberosClient implements java.io.Closeable {
public void setHiveClient(HiveClient client) {
this.hiveClient = client;
}
+
+ /**
+ * Returns the real (non-proxy) {@link UserGroupInformation} that was
obtained after Kerberos
+ * login via keytab. Used by callers that need to bind the JAAS Subject to
the current thread
+ * (e.g., via {@code ugi.doAs(...)}) before performing a Kerberos-protected
RPC call.
+ *
+ * @return the real login UGI.
+ * @throws IllegalStateException if {@link #login()} has not been called yet.
+ */
+ public UserGroupInformation getRealLoginUgi() {
+ Preconditions.checkState(realLoginUgi != null, "KerberosClient.login() has
not been called");
+ return realLoginUgi;
+ }
}
diff --git
a/catalogs/hive-metastore-common/src/test/java/org/apache/gravitino/hive/client/TestHive2HMSWithKerberosNoImpersonation.java
b/catalogs/hive-metastore-common/src/test/java/org/apache/gravitino/hive/client/TestHive2HMSWithKerberosNoImpersonation.java
new file mode 100644
index 0000000000..2e12c966b3
--- /dev/null
+++
b/catalogs/hive-metastore-common/src/test/java/org/apache/gravitino/hive/client/TestHive2HMSWithKerberosNoImpersonation.java
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.gravitino.hive.client;
+
+import java.util.Properties;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.TestInstance;
+
+/**
+ * Integration test for Kerberos-enabled Hive2 HMS with impersonation
<b>disabled</b>.
+ *
+ * <p>This test covers the fix for the non-impersonation Kerberos path in
{@link HiveClientFactory}:
+ * the path was missing a {@code realUgi.doAs()} wrapper, causing GSSAPI to
fail with "No valid
+ * credentials provided" because the JAAS Subject (containing the TGT) was
never bound to the
+ * current thread.
+ *
+ * <p>The existing {@link TestHive2HMSWithKerberos} only tests {@code
impersonation=true}. This
+ * class reuses the same Docker KDC infrastructure but overrides {@link
#createHiveProperties()} to
+ * set {@code authentication.impersonation-enable=false}, exercising the
previously broken code
+ * path. {@link TestHive2HMSWithKerberos#startHiveContainer()} calls {@link
#createHiveProperties()}
+ * via polymorphism, so no {@code startHiveContainer} override is needed here.
+ *
+ * <p>Requires Docker; run with {@code -PskipDockerTests=false}.
+ */
+@Tag("gravitino-docker-test")
+@TestInstance(TestInstance.Lifecycle.PER_CLASS)
+public class TestHive2HMSWithKerberosNoImpersonation extends
TestHive2HMSWithKerberos {
+
+ /**
+ * Override to disable impersonation. All other Kerberos properties
(principal, keytab, sasl) are
+ * inherited from {@link TestHive2HMSWithKerberos#createHiveProperties()}.
+ *
+ * <p>Setting {@code authentication.impersonation-enable=false} forces
{@link HiveClientFactory}
+ * to take the non-impersonation Kerberos branch in {@code
createHiveClientInternal()}.
+ */
+ @Override
+ protected Properties createHiveProperties() {
+ Properties properties = super.createHiveProperties();
+ properties.setProperty("authentication.impersonation-enable", "false");
+ return properties;
+ }
+}
