Indhumathi Muthumurugesh created HIVE-29753:
-----------------------------------------------
Summary: StorageBasedAuthorizationProvider silently blocks on slow
HDFS getFileInfo calls with no observable logging
Key: HIVE-29753
URL: https://issues.apache.org/jira/browse/HIVE-29753
Project: Hive
Issue Type: Improvement
Components: Authorization, Metastore
Reporter: Indhumathi Muthumurugesh
When StorageBasedAuthorizationProvider.checkPermissions() calls
FileUtils.getFileStatusOrNull() to resolve HDFS path permissions, the
underlying NameNode RPC (getFileInfo) can block for an extended duration.
During this time, Metastore Handler threads are silently stuck with no INFO or
WARN log emitted — the issue is only detectable via thread dumps or DEBUG-level
logging.
Example thread dump showing the silent block:
{code:java}
java.lang.Thread.State: WAITING (on object monitor)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1571)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(...)
at
org.apache.hadoop.hive.common.FileUtils.getFileStatusOrNull(FileUtils.java:1114)
at
org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider.checkPermissions(...)
at org.apache.hadoop.hive.metastore.HMSHandler.get_database(...){code}
In production this can manifest as Metastore thread pool exhaustion with no
actionable log evidence.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
