[
https://issues.apache.org/jira/browse/SPARK-7116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14512908#comment-14512908
]
Joseph K. Bradley commented on SPARK-7116:
------------------------------------------
[~davies] Is there any good way to fix this? It looks like no action has been
performed on the persisted RDD's children before the method exits, so
unpersisting might not be a good idea.
> Intermediate RDD cached but never unpersisted
> ---------------------------------------------
>
> Key: SPARK-7116
> URL: https://issues.apache.org/jira/browse/SPARK-7116
> Project: Spark
> Issue Type: Improvement
> Components: PySpark, SQL
> Affects Versions: 1.3.1
> Reporter: Kalle Jepsen
>
> In
> https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/pythonUdfs.scala#L233
> an intermediate RDD is cached, but never unpersisted. It shows up in the
> 'Storage' section of the Web UI, but cannot be removed. There's already a
> comment in the source, suggesting to 'clean up'. If that cleanup is more
> involved than simply calling `unpersist`, it probably exceeds my current
> Scala skills.
> Why that is a problem:
> I'm adding a constant column to a DataFrame of about 20M records resulting
> from an inner join with {{df.withColumn(colname, ud_func())}} , where
> {{ud_func}} is simply a wrapped {{lambda: 1}}. Before and after applying the
> UDF the DataFrame takes up ~430MB in the cache. The cached intermediate RDD
> however takes up ~10GB(!) of storage, and I know of no way to uncache it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
