[
https://issues.apache.org/jira/browse/SPARK-22793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16292174#comment-16292174
]
zuotingbing commented on SPARK-22793:
-------------------------------------
{code:java}
lazy val metadataHive: HiveClient = sharedState.metadataHive.newSession()
{code}
HiveClient has been created by
{code:java}
sharedState.metadataHive
{code}
but will be created again in
{code:java}
.newSession()
{code}
> Memory leak in Spark Thrift Server
> ----------------------------------
>
> Key: SPARK-22793
> URL: https://issues.apache.org/jira/browse/SPARK-22793
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.0.2
> Reporter: zuotingbing
> Priority: Critical
>
> 1. Start HiveThriftServer2.
> 2. Connect to thriftserver through beeline.
> 3. Close the beeline.
> 4. repeat step2 and step 3 for several times, which caused the leak of Memory.
> we found there are many directories never be dropped under the path
> {code:java}
> hive.exec.local.scratchdir
> {code} and
> {code:java}
> hive.exec.scratchdir
> {code} , as we know the scratchdir has been added to deleteOnExit when it be
> created. So it means that the cache size of FileSystem deleteOnExit will keep
> increasing until JVM terminated.
> In addition, we use
> {code:java}
> jmap -histo:live [PID]
> {code} to printout the size of objects in HiveThriftServer2 Process, we can
> find the object "org.apache.spark.sql.hive.client.HiveClientImpl" and
> "org.apache.hadoop.hive.ql.session.SessionState" keep increasing even though
> we closed all the beeline connections, which caused the leak of Memory.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
