[
https://issues.apache.org/jira/browse/HIVE-11499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vaibhav Gumashta updated HIVE-11499:
------------------------------------
Description: When UDFs are used, we create a new classloader to add the UDF
jar. Similar to what hadoop's reflection utils does(HIVE-11408), datanucleus
caches the classloaders
(https://github.com/datanucleus/datanucleus-core/blob/3.2/src/java/org/datanucleus/NucleusContext.java#L161).
JDOPersistanceManager factory (1 per JVM) holds on to a NucleusContext
reference
(https://github.com/datanucleus/datanucleus-api-jdo/blob/3.2/src/java/org/datanucleus/api/jdo/JDOPersistenceManagerFactory.java#L115).
Until we call NucleusContext#close, the classloader cache is not cleared. In
case of UDFs this can lead to permgen leak, as shown in the attached
screenshot, where NucleusContext holds on to several URLClassloader objects.
(was: When UDFs are used, we create a new classloader to add the UDF jar.
Similar to what hadoop's reflection utils does(HIVE-11408), datanucleus caches
the classloaders
(https://github.com/datanucleus/datanucleus-core/blob/3.2/src/java/org/datanucleus/NucleusContext.java#L161).
JDOPersistanceManager factory (1 per JVM) holds on to a NucleusContext
reference
(https://github.com/datanucleus/datanucleus-api-jdo/blob/3.2/src/java/org/datanucleus/api/jdo/JDOPersistenceManagerFactory.java#L115).
Until we call NucleusContext#close, the classloader cache is not cleared. In
case of UDFs this can lead as shows in the attached screenshot, where
NucleusContext holds on to several URLClassloader objects.)
> Datanucleus leaks classloaders when used using embedded metastore with
> HiveServer2 with UDFs
> --------------------------------------------------------------------------------------------
>
> Key: HIVE-11499
> URL: https://issues.apache.org/jira/browse/HIVE-11499
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Metastore
> Affects Versions: 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.1.1, 1.2.1
> Reporter: Vaibhav Gumashta
> Assignee: Vaibhav Gumashta
> Attachments: HS2-NucleusCache-Leak.tiff
>
>
> When UDFs are used, we create a new classloader to add the UDF jar. Similar
> to what hadoop's reflection utils does(HIVE-11408), datanucleus caches the
> classloaders
> (https://github.com/datanucleus/datanucleus-core/blob/3.2/src/java/org/datanucleus/NucleusContext.java#L161).
> JDOPersistanceManager factory (1 per JVM) holds on to a NucleusContext
> reference
> (https://github.com/datanucleus/datanucleus-api-jdo/blob/3.2/src/java/org/datanucleus/api/jdo/JDOPersistenceManagerFactory.java#L115).
> Until we call NucleusContext#close, the classloader cache is not cleared.
> In case of UDFs this can lead to permgen leak, as shown in the attached
> screenshot, where NucleusContext holds on to several URLClassloader objects.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
