JoegenUSTC opened a new issue, #11353:
URL: https://github.com/apache/gravitino/issues/11353
### Environment
- Gravitino version: main / v0.9.x (confirmed on v1.2.0-rc8)
- Hive Metastore version: HMS with Kerberos authentication
- Configuration: `hive.metastore.sasl.enabled=true`, impersonation
**disabled**
### Problem
When a Hive catalog is configured with Kerberos authentication and
impersonation is **not** enabled, connecting to HMS fails with:
```
GSS initiate failed
No valid credentials provided (Mechanism level: Failed to find any Kerberos
tgt)
```
or
```
java.lang.NoSuchMethodException: HiveClientImpl.<init>(HiveVersion,
Properties)
```
Root cause analysis reveals **three independent bugs** in
`HiveClientFactory`, all in the same code path.
---
### Bug 1 — `baseLoader` uses TCCL, causing UGI class to be loaded twice
**Location**: `HiveClientFactory.java`, `createHiveClientWithBackend()`,
lines 121 and 143
**Code**:
```java
// Current (buggy)
classloader = HiveClientClassLoader.createLoader(HIVE3,
Thread.currentThread().getContextClassLoader()); // TCCL is not stable
```
`HiveClientClassLoader` uses `baseLoader` to determine which classes are
"shared" (delegated upward via `isSharedClass`). This includes
`UserGroupInformation` (UGI) and the `HiveClient` interface.
Because `Thread.currentThread().getContextClassLoader()` (TCCL) is **not
stable** across threads:
- `KerberosClient.login()` runs on thread A → UGI loaded by TCCL-A → TGT
stored in TCCL-A's UGI static state
- HMS connection runs on thread B → UGI loaded by TCCL-B → different `Class`
object → static state is empty → TGT not found → Kerberos handshake fails
The same `UserGroupInformation` class ends up loaded by two different
ClassLoaders, so their static state is not shared.
**Fix**: Use `HiveClientFactory.class.getClassLoader()` as `baseLoader` — it
is stable and consistent:
```java
ClassLoader factoryCl = HiveClientFactory.class.getClassLoader();
classloader = HiveClientClassLoader.createLoader(HIVE3, factoryCl);
```
---
### Bug 2 — `HiveVersion` enum type mismatch causes `NoSuchMethodException`
**Location**: `HiveClientFactory.java`, `createHiveClientImpl()`, lines
165–167
**Code**:
```java
// Current (buggy)
Constructor<?> hiveClientImplCtor = hiveClientImplClass.getConstructor(
HiveClientClassLoader.HiveVersion.class, // ← system CL's HiveVersion
Properties.class);
```
`hiveClientImplClass` is loaded from the **isolated**
`HiveClientClassLoader` (it is a barrier class redefined via `defineClass`).
Its constructor expects a `HiveVersion` parameter belonging to the **isolated**
ClassLoader's type space.
Passing `HiveClientClassLoader.HiveVersion.class` (system CL) causes
`getConstructor` to find no matching constructor → `NoSuchMethodException`.
**Fix**: Load `HiveVersion` from the isolated ClassLoader before
constructing:
```java
Class<?> hiveVersionInIsolated =
classloader.loadClass(HiveClientClassLoader.HiveVersion.class.getName());
Object isolatedVersion =
Enum.valueOf((Class<? extends Enum>) hiveVersionInIsolated,
version.name());
Constructor<?> ctor =
hiveClientImplClass.getConstructor(hiveVersionInIsolated,
Properties.class);
return (HiveClient) ctor.newInstance(isolatedVersion, properties);
```
---
### Bug 3 — Non-impersonation Kerberos path never binds the JAAS Subject to
the current thread
**Location**: `HiveClientFactory.java`, `createHiveClientInternal()`, lines
205–206
**Code**:
```java
// Current (buggy) — non-impersonation branch
} else {
return createHiveClientImpl(classloader.getHiveVersion(), properties,
classloader);
}
```
`KerberosClient.login()` calls
`UserGroupInformation.loginUserFromKeytab(...)`, which stores the TGT in
`loginUser.subject`. However, the HMS Thrift client performs a SASL/GSSAPI
handshake at connection time. GSSAPI looks for Kerberos credentials in the
**current thread's JAAS Subject context** (`AccessControlContext`). Since
`javax.security.auth.useSubjectCredsOnly=true` by default in the JVM, if no
Subject is bound to the current thread, GSSAPI immediately fails.
`loginUser.subject` is **not** automatically bound to the current thread. It
must be explicitly injected via `Subject.doAs(subject, action)` or
`ugi.doAs(action)`.
The impersonation path (`createProxyHiveClientImpl`) correctly uses
`ugi.doAs(...)` — the non-impersonation Kerberos path is missing the equivalent.
**Fix**:
```java
} else if (enableKerberos) {
UserGroupInformation realUgi = kerberosClient.getRealLoginUgi();
final HiveClientClassLoader.HiveVersion hiveVersion =
classloader.getHiveVersion();
return realUgi.doAs(
(PrivilegedExceptionAction<HiveClient>)
() -> createHiveClientImpl(hiveVersion, properties,
classloader));
}
```
This is consistent with how `ImpalaEngineAdapter` and `ProxyHiveClientImpl`
already handle the Kerberos + `doAs` pattern.
---
### Summary
| # | Root cause | Effect | Fix |
|---|-----------|--------|-----|
| 1 | `baseLoader` uses unstable TCCL | UGI loaded twice; TGT not shared
across threads | Use `HiveClientFactory.class.getClassLoader()` |
| 2 | `getConstructor` passes system CL's `HiveVersion` |
`NoSuchMethodException` on every `createHiveClientImpl` call | Load
`HiveVersion` from isolated CL |
| 3 | Non-impersonation Kerberos path missing `doAs` | GSSAPI can't find TGT
→ `No valid credentials` | Wrap in `realUgi.doAs(...)` |
All three bugs exist in the current `main` branch. I am happy to submit a PR
with the fixes and corresponding tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
