[
https://issues.apache.org/jira/browse/SPARK-22793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zuotingbing updated SPARK-22793:
--------------------------------
Description:
1. Start HiveThriftServer2.
2. Connect to thriftserver through beeline.
3. Close the beeline.
4. repeat step2 and step 3 for several times, which caused the leak of Memory.
we found there are many directories never be dropped under the path
{code:java}
hive.exec.local.scratchdir
{code} and
{code:java}
hive.exec.scratchdir
{code} , as we know the scratchdir has been added to deleteOnExit when it be
created. So it means that the cache size of FileSystem deleteOnExit will keep
increasing until JVM terminated.
In addition, we use
{code:java}
jmap -histo:live [PID]
{code} to printout the size of objects in HiveThriftServer2 Process, we can
find the object "org.apache.spark.sql.hive.client.HiveClientImpl" and
"org.apache.hadoop.hive.ql.session.SessionState" keep increasing even though we
closed all the beeline connections, which caused the leak of Memory.
was:
1. Start HiveThriftServer2
2. Connect to thriftserver through beeline
3. Close the beeline
4. repeat step2 and step 3 for several times
we found there are many directories never be dropped under the path
{code:java}
hive.exec.local.scratchdir
{code} and
{code:java}
hive.exec.scratchdir
{code} , as we know the scratchdir has been added to deleteOnExit when it be
created. So it means that the cache size of FileSystem deleteOnExit will keep
increasing until JVM terminated.
In addition, we use
{code:java}
jmap -histo:live [PID]
{code} to printout the size of objects in HiveThriftServer2 Process, we can
find the object "org.apache.spark.sql.hive.client.HiveClientImpl" and
"org.apache.hadoop.hive.ql.session.SessionState" keep increasing even though we
closed all the beeline connections, which caused the leak of Memory.
> Memory leak in Spark Thrift Server
> ----------------------------------
>
> Key: SPARK-22793
> URL: https://issues.apache.org/jira/browse/SPARK-22793
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.0.2
> Reporter: zuotingbing
> Priority: Critical
>
> 1. Start HiveThriftServer2.
> 2. Connect to thriftserver through beeline.
> 3. Close the beeline.
> 4. repeat step2 and step 3 for several times, which caused the leak of Memory.
> we found there are many directories never be dropped under the path
> {code:java}
> hive.exec.local.scratchdir
> {code} and
> {code:java}
> hive.exec.scratchdir
> {code} , as we know the scratchdir has been added to deleteOnExit when it be
> created. So it means that the cache size of FileSystem deleteOnExit will keep
> increasing until JVM terminated.
> In addition, we use
> {code:java}
> jmap -histo:live [PID]
> {code} to printout the size of objects in HiveThriftServer2 Process, we can
> find the object "org.apache.spark.sql.hive.client.HiveClientImpl" and
> "org.apache.hadoop.hive.ql.session.SessionState" keep increasing even though
> we closed all the beeline connections, which caused the leak of Memory.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
