[
https://issues.apache.org/jira/browse/SPARK-14261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15275238#comment-15275238
]
Weizhong edited comment on SPARK-14261 at 5/10/16 6:52 AM:
-----------------------------------------------------------
I also face this issue.
I found each session will add one HiveConf on sun.misc.Launcher$AppClassLoader
and one on org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1. All
these HiveConf can't be released until OOM.
>From the OOM dump(Use Eclipse Memory Analyzer to analyze), all these HiveConf
>have ref, it's GC root like below:
{noformat}
org.apache.hadoop.hive.conf.HiveConf
conf org.apache.hadoop.hive.ql.session.SessionState$SessionStates
value java.lang.ThreadLocal$ThreadLocalMap$Entry
[19] java.lang.ThreadLocal$ThreadLocalMap$Entry[32]
table java.lang.ThreadLocal$ThreadLocalMap
threadLocals java.lang.Thread
referent java.util.WeakHashMap$Entry
conf org.apache.hadoop.hive.ql.session.SessionState
state org.apache.spark.sql.hive.client.ClientWrapper
metaHive, metadataHive, metaHive
org.apache.spark.sql.hive.client.HiveContext
$outer org.apache.spark.sql.SQLContext$$anon$4
[265] java.lang.Object[267]
array java.util.concurrent.CopyWriteArrayList
listeners org.apache.spark.scheduler.LiveListenerBus
$outer org.apache.spark.util.AsynchronousListenerBus$$anon$1
{noformat}
was (Author: sephiroth-lin):
I also face this issue.
I found every session will add one metastore connection, and when i dump the
heap, i found have many Hive and HiveConf object don't be removed
> Memory leak in Spark Thrift Server
> ----------------------------------
>
> Key: SPARK-14261
> URL: https://issues.apache.org/jira/browse/SPARK-14261
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.6.0
> Reporter: Xiaochun Liang
> Attachments: 16716_heapdump_64g.PNG, 16716_heapdump_80g.PNG,
> MemorySnapshot.PNG
>
>
> I am running Spark Thrift server on Windows Server 2012. The Spark Thrift
> server is launched as Yarn client mode. Its memory usage is increased
> gradually with the queries in. I am wondering there is memory leak in Spark
> Thrift server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
