skywalker0618 opened a new issue, #19321:
URL: https://github.com/apache/hudi/issues/19321
### Bug Description
**What happened:**
`HoodieHiveSyncClient` (hiveql sync mode) leaks one Hive Metastore
connection on every sync cycle whenever the underlying metastore client is
rebuilt by `RetryingMetaStoreClient`. For long-running jobs that sync on every
commit (Flink streaming, Spark structured streaming, DeltaStreamer continuous
mode), the metastore's open-connection count grows monotonically until HMS hits
its connection limit.
### Root cause
`HoodieHiveSyncClient` holds its metastore handle in a single field:
private IMetaStoreClient client; // a RetryingMetaStoreClient proxy
`RetryingMetaStoreClient` is a dynamic proxy: when a call throws a transient
`TException`/`TTransportException` (e.g. `"SASL authentication not complete"`,
or any dropped/reset connection), it **transparently tears down the current
underlying `IMetaStoreClient`, builds a fresh one, and retries the call.** The
retry succeeds, so the sync itself looks healthy in the logs.
The problem is in teardown. `close()` only calls `Hive.closeCurrent()`:
public void close() {
...
if (client != null) {
Hive.closeCurrent(); // closes the MSC bound to the thread-local
Hive singleton
client = null;
}
}
`Hive.closeCurrent()` closes the metastore client referenced by the
thread-local `Hive` singleton. But once `RetryingMetaStoreClient` has rebuilt
the connection, the **freshly-built** underlying MSC is reachable only through
the proxy chain on this `client` field — the `Hive` singleton still points at
the **older** MSC instance. So `Hive.closeCurrent()`:
- closes the *older* MSC (already dead/replaced), and
- **orphans the retry-created MSC**, whose live socket to HMS is never
closed.
Result: one leaked TCP connection to the metastore per cycle in which a
retry occurred.
### Why it's easy to miss
- The sync succeeds (retry masks the transient failure), so there's no error
surfaced.
- The leak is only visible as a slowly climbing HMS `current connections` /
active-thrift-connection count over hours/days, eventually manifesting as HMS
connection exhaustion or `Too many open files` on the client.
**What you expected:**
`close()` should reliably release the metastore connection that is actually
live on `this.client`, regardless of whether `RetryingMetaStoreClient` has
swapped the underlying instance out from under the `Hive` singleton.
Fix: close the proxied `IMetaStoreClient` **directly** before falling back
to `Hive.closeCurrent()`. `client.close()` walks the proxy and closes the
currently-bound real MSC by identity, so the retry-created connection is always
released; `Hive.closeCurrent()` stays as a belt-and-suspenders cleanup for the
singleton path. Wrap the direct close in its own try/catch so a transient close
failure still lets the fallback run.
if (client != null) {
try {
client.close(); // closes the currently-bound MSC
(incl. retry-rebuilt one)
} catch (Exception e) {
LOG.warn("Failed to close IMetaStoreClient proxy directly; "
+ "Hive.closeCurrent() will run anyway", e);
}
Hive.closeCurrent(); // fallback for the thread-local Hive
singleton
client = null;
}
Expected outcome: HMS open-connection count stays bounded across sync cycles
(returns to baseline between cycles) instead of growing by one per retry cycle.
**Steps to reproduce:**
1.
2.
3.
### Environment
- Hudi: 0.14.x
- Sync mode: `hoodie.datasource.hive_sync.mode=hiveql` (uses
`IMetaStoreClient` + `HiveQueryDDLExecutor`)
- Any deployment where the metastore client periodically retries (transient
SASL/connection resets), which is common under Kerberos/SASL.
- Highest impact: long-running jobs that sync every commit (Flink streaming,
DeltaStreamer continuous mode).
### Logs and Stack Trace
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
