[jira] [Comment Edited] (SPARK-22793) Memory leak in Spark Thrift Server
[
https://issues.apache.org/jira/browse/SPARK-22793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16292314#comment-16292314
]
zuotingbing edited comment on SPARK-22793 at 12/26/17 2:00 AM:
---
yes the master branch also has this problem.
was (Author: zuo.tingbing9):
yes the master branch also has this problem, but the difference is so big
between branch master and 2.0 . Could someone help to merge this to the master
branch?
> Memory leak in Spark Thrift Server
> --
>
> Key: SPARK-22793
> URL: https://issues.apache.org/jira/browse/SPARK-22793
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.0.2, 2.2.1
>Reporter: zuotingbing
>Priority: Critical
>
> 1. Start HiveThriftServer2.
> 2. Connect to thriftserver through beeline.
> 3. Close the beeline.
> 4. repeat step2 and step 3 for several times, which caused the leak of Memory.
> we found there are many directories never be dropped under the path
> {code:java}
> hive.exec.local.scratchdir
> {code} and
> {code:java}
> hive.exec.scratchdir
> {code} , as we know the scratchdir has been added to deleteOnExit when it be
> created. So it means that the cache size of FileSystem deleteOnExit will keep
> increasing until JVM terminated.
> In addition, we use
> {code:java}
> jmap -histo:live [PID]
> {code} to printout the size of objects in HiveThriftServer2 Process, we can
> find the object "org.apache.spark.sql.hive.client.HiveClientImpl" and
> "org.apache.hadoop.hive.ql.session.SessionState" keep increasing even though
> we closed all the beeline connections, which caused the leak of Memory.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-22793) Memory leak in Spark Thrift Server
[
https://issues.apache.org/jira/browse/SPARK-22793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16292314#comment-16292314
]
zuotingbing edited comment on SPARK-22793 at 12/15/17 10:48 AM:
yes the master branch also has this problem, but the difference is so big
between branch master and 2.0 . Could someone help to merge this to the master
branch?
was (Author: zuo.tingbing9):
yes the master branch also has this problem,but the different is so big between
branch master to 2.0 . i am not sure this can be merged to the master branch.
> Memory leak in Spark Thrift Server
> --
>
> Key: SPARK-22793
> URL: https://issues.apache.org/jira/browse/SPARK-22793
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.0.2
>Reporter: zuotingbing
>Priority: Critical
>
> 1. Start HiveThriftServer2.
> 2. Connect to thriftserver through beeline.
> 3. Close the beeline.
> 4. repeat step2 and step 3 for several times, which caused the leak of Memory.
> we found there are many directories never be dropped under the path
> {code:java}
> hive.exec.local.scratchdir
> {code} and
> {code:java}
> hive.exec.scratchdir
> {code} , as we know the scratchdir has been added to deleteOnExit when it be
> created. So it means that the cache size of FileSystem deleteOnExit will keep
> increasing until JVM terminated.
> In addition, we use
> {code:java}
> jmap -histo:live [PID]
> {code} to printout the size of objects in HiveThriftServer2 Process, we can
> find the object "org.apache.spark.sql.hive.client.HiveClientImpl" and
> "org.apache.hadoop.hive.ql.session.SessionState" keep increasing even though
> we closed all the beeline connections, which caused the leak of Memory.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
