[
https://issues.apache.org/jira/browse/SPARK-24437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16671686#comment-16671686
]
David Vogelbacher commented on SPARK-24437:
-------------------------------------------
Hey [~mgaido], I am seeing something similar in one of our long running
applications.
The longer it runs the higher heap usage in the driver grows. Taking a heap
dump, most of its space is taken up by {{UnsafeHashedRelation}} instances.
Looking at their paths to GC, they are strongly referenced by
{{InMemoryRelation}} instances:
!Screen Shot 2018-11-01 at 10.38.30 AM.png!
So it looks like if dataframes are cached the {{UnsafeHashedRelation}}s are
never cleaned up, because they are strongly references by generated codegen
plan?
The context cleaner can only clean them up when they are no longer strongly
referenced.
> Memory leak in UnsafeHashedRelation
> -----------------------------------
>
> Key: SPARK-24437
> URL: https://issues.apache.org/jira/browse/SPARK-24437
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.2.0
> Reporter: gagan taneja
> Priority: Critical
> Attachments: Screen Shot 2018-05-30 at 2.05.40 PM.png, Screen Shot
> 2018-05-30 at 2.07.22 PM.png, Screen Shot 2018-11-01 at 10.38.30 AM.png
>
>
> There seems to memory leak withÂ
> org.apache.spark.sql.execution.joins.UnsafeHashedRelation
> We have a long running instance of STS.
> With each query execution requiring Broadcast Join, UnsafeHashedRelation is
> getting added for cleanup in ContextCleaner. This reference of
> UnsafeHashedRelation is being held at some other Collection and not becoming
> eligible for GC and because of this ContextCleaner is not able to clean it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
