skywalker0618 opened a new pull request, #20219: URL: https://github.com/apache/hudi/pull/20219
### Describe the issue this Pull Request addresses In HiveQL sync mode, every sync closes its own metastore client partway through and only survives on `RetryingMetaStoreClient`'s single reconnect. `HoodieHiveSyncClient` takes its metastore client from the `Hive` instance Hive caches for the thread (`IMetaStoreClientUtil.getMSC`). `HiveQueryDDLExecutor` then starts a Hive `SessionState` for its Driver. Setting up that session's authorization (with the default `hive.security.authorization.manager`, `SQLStdHiveAuthorizerFactory`) sets `hive.metastore.filter.hook` to `AuthorizationMetaStoreFilterHook` and calls `Hive.getWithoutRegisterFns(sessionConf)`. That finds the cached instance's metastore conf different, so Hive replaces the instance and closes its client, which is the one the sync client holds. The sync's next metastore call fails and is retried: ``` INFO hive.metastore - Mestastore configuration hive.metastore.filter.hook changed from org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl to org.apache.hadoop.hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook INFO hive.metastore - Closed a connection to metastore, current connections: 0 ... WARN org.apache.hadoop.hive.metastore.RetryingMetaStoreClient - MetaStoreClient lost connection. Attempting to reconnect (1 of 1) after 1s. listPartitions org.apache.thrift.transport.TTransportException: SASL authentication not complete ``` Each sync pays a 1s sleep and an extra connection, and it fails outright whenever that one reconnect does not succeed, such as during a brief metastore hiccup. ### Summary and Changelog The sync client keeps its metastore client to itself, so a Hive session on the same thread no longer closes it. - `HoodieHiveSyncClient.createMetaStoreClient()` takes the instance that owns the client out of Hive's thread cache with `Hive.set(null)`. A Hive session on the thread then builds an instance of its own instead of replacing this one. `Hive.closeCurrent()` cannot be used for this, since it closes the instance's client too. - `HoodieHiveSyncClient.close()` already closes the client directly; its comment is updated to match. - New test `TestHiveSyncTool#testHiveQLSyncKeepsItsMetastoreClientOpen` syncs in HiveQL mode with a client conf (default filter hook, authorization not yet applied) and no metastore retries, so a closed client fails the sync. It fails without the fix with `TTransportException: Cannot write to null outputStream`. ### Impact HiveQL-mode syncs no longer lose their metastore connection, so they skip the reconnect and its 1s sleep and no longer fail when the reconnect does. No API or config change. ### Risk Level low. The client is created the same way; only its place in Hive's thread cache changes. The instance a Hive session leaves in the cache is still closed by `HiveQueryDDLExecutor.close()` and `HoodieHiveSyncClient.close()` as before. The full `hudi-hive-sync` test suite passes. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
