[
https://issues.apache.org/jira/browse/SPARK-14261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15219304#comment-15219304
]
Xiaochun Liang commented on SPARK-14261:
----------------------------------------
I did take heap dump when the serer is running. Unfortunately the heap dump
file is too big to be uploaded, I just uploaded the screenshots of the heap
dump.
Some explanations on the screenshots of heap dump:
1. The heap dump file was analyzed by MemoryAnalyzer,
2. 16716 is the process id,
3. 64g means the dump file was generated when the java process reaches 6.4g,
while 80g means the dump file was generated when the java process reaches 8.0g
I looked through the memory under two cases and had following findings:
1. java.io.DeleteOnExitHook took more memories in both cases,
2. The memory of org.apache.hadoop.hive.ql.processors.CommandProcessorFactory
increases a lot when memory reaches 8g,
3. The memory of org.apache.hadoop.conf.Configuration increases a lot when
memory reaches 8g.
> Memory leak in Spark Thrift Server
> ----------------------------------
>
> Key: SPARK-14261
> URL: https://issues.apache.org/jira/browse/SPARK-14261
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.6.0
> Reporter: Xiaochun Liang
> Attachments: 16716_heapdump_64g.PNG, 16716_heapdump_80g.PNG,
> MemorySnapshot.PNG
>
>
> I am running Spark Thrift server on Windows Server 2012. The Spark Thrift
> server is launched as Yarn client mode. Its memory usage is increased
> gradually with the queries in. I am wondering there is memory leak in Spark
> Thrift server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
