[
https://issues.apache.org/jira/browse/HIVE-26376?focusedWorklogId=788922&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-788922
]
ASF GitHub Bot logged work on HIVE-26376:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 08/Jul/22 10:26
Start Date: 08/Jul/22 10:26
Worklog Time Spent: 10m
Work Description: zabetak opened a new pull request, #3424:
URL: https://github.com/apache/hive/pull/3424
### What changes were proposed in this pull request?
Close filesystem references in StorageBasedAuthorizationProvider after use.
### Why are the changes needed?
To prevent leaving filesystem objects and hitting OOM error.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Manual tests.
Issue Time Tracking
-------------------
Worklog Id: (was: 788922)
Remaining Estimate: 0h
Time Spent: 10m
> Hive Metastore connection leak (OOM Error)
> ------------------------------------------
>
> Key: HIVE-26376
> URL: https://issues.apache.org/jira/browse/HIVE-26376
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 3.1.2
> Environment: !Screenshot 2022-07-07 at 11.52.33 AM.png!
> Reporter: Ranith Sardar
> Assignee: Stamatis Zampetakis
> Priority: Major
> Attachments: Screenshot 2022-07-07 at 11.52.33 AM.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive version:3.1.2
> Hive metastore heap size is 14GB, Memory Leak is happening after 4-5 days,
> hive meta-store throwing error with OOM.
> If we disable the configuration, the memory leak disappears.
> In the case of, Heap dump size 3.5GB, a large number of filesystem objects(>
> 9k instances) are being retained. It's occupying most of the heap space.
> Added snapshot from the eclipse MAT.
> Bellow are part of the stack trace for OOM error:
> {code:java}
> at
> org.apache.hadoop.hive.common.FileUtils.getFileStatusOrNull(Lorg/apache/hadoop/fs/FileSystem;Lorg/apache/hadoop/fs/Path;)Lorg/apache/hadoop/fs/FileStatus;
> (FileUtils.java:801)
> at
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider.checkPermissions(Lorg/apache/hadoop/conf/Configuration;Lorg/apache/hadoop/fs/Path;Ljava/util/EnumSet;)V
> (StorageBasedAuthorizationProvider.java:371)
> at
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider.authorize(Lorg/apache/hadoop/fs/Path;[Lorg/apache/hadoop/hive/ql/security/authorization/Privilege;[Lorg/apache/hadoop/hive/ql/security/authorization/Privilege;)V
> (StorageBasedAuthorizationProvider.java:346)
> at
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider.authorize(Lorg/apache/hadoop/hive/metastore/api/Database;[Lorg/apache/hadoop/hive/ql/security/authorization/Privilege;[Lorg/apache/hadoop/hive/ql/security/authorization/Privilege;)V
> (StorageBasedAuthorizationProvider.java:154)
> at
> org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener.authorizeReadDatabase(Lorg/apache/hadoop/hive/metastore/events/PreReadDatabaseEvent;)V
> (AuthorizationPreEventListener.java:208)
> at
> org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener.onEvent(Lorg/apache/hadoop/hive/metastore/events/PreEventContext;)V
> (AuthorizationPreEventListener.java:153)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.firePreEvent(Lorg/apache/hadoop/hive/metastore/events/PreEventContext;)V
> (HiveMetaStore.java:3221)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_database(Ljava/lang/String;)Lorg/apache/hadoop/hive/metastore/api/Database;
> (HiveMetaStore.java:1352){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
