[
https://issues.apache.org/jira/browse/HIVE-16160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15904036#comment-15904036
]
Sushanth Sowmyan commented on HIVE-16160:
-----------------------------------------
Looking at the heap dump that Kavan provided me for this, it appears we have a
large number of leftover objects in a DN WeakValueMap from
ClassLoaderResolverImpl, and is an issue similar to HIVE-11499.
The underlying issue is not that there is necessarily a leak as much as that
the ClassLoaderResolverImpl winds up having a WeakValueMap<String,Class>, where
all the values are stored as WeakReferences with a ReferenceQueue tracking
expiry. Thus, at load time, after it's done loading all the classes, the key
list alone is enough for us to wind up going to about 200megs, even if all the
values are GCed, because the keys are not removed till after a reap() is called
on the WeakValueMap, which does not wind up getting called because we don't use
that map after a while. More to the point, the map that is large is the
"unloadedClasses" map of the ClassLoaderResolverImpl.
I'll upload more details (screenshots of the memory analysis at least, since
the dump itself is huge), and a resolution soon.
> OutOfMemoryError: GC overhead limit exceeded on Hiveserver2
> ------------------------------------------------------------
>
> Key: HIVE-16160
> URL: https://issues.apache.org/jira/browse/HIVE-16160
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
> Reporter: Kavan Suresh
> Assignee: Sushanth Sowmyan
> Priority: Critical
>
> Hs2 process killed by OOM:
> {code:java}
> ERROR [HiveServer2-Handler-Pool: Thread-1361771]:
> metastore.RetryingHMSHandler (RetryingHMSHandler.java:invokeInternal(203)) -
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.lang.String.toLowerCase(String.java:2647)
> at
> org.datanucleus.Configuration.getInternalNameForProperty(Configuration.java:188)
> at
> org.datanucleus.ExecutionContextImpl.getProperty(ExecutionContextImpl.java:1012)
> at
> org.datanucleus.state.AbstractStateManager.updateLevel2CacheForFields(AbstractStateManager.java:979)
> at
> org.datanucleus.state.AbstractStateManager.loadFieldsInFetchPlan(AbstractStateManager.java:1097)
> at
> org.datanucleus.ExecutionContextImpl.performDetachAllOnTxnEndPreparation(ExecutionContextImpl.java:4544)
> at
> org.datanucleus.ExecutionContextImpl.preCommit(ExecutionContextImpl.java:4199)
> at
> org.datanucleus.ExecutionContextImpl.transactionPreCommit(ExecutionContextImpl.java:770)
> at
> org.datanucleus.TransactionImpl.internalPreCommit(TransactionImpl.java:385)
> at org.datanucleus.TransactionImpl.commit(TransactionImpl.java:275)
> at
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:107)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.commitTransaction(ObjectStore.java:570)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:1033)
> at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:103)
> at com.sun.proxy.$Proxy7.getTable(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_core(HiveMetaStore.java:1915)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:1887)
> at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
> at com.sun.proxy.$Proxy12.get_table(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1271)
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getTable(SessionHiveMetaStoreClient.java:131)
> at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:178)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
