R created SPARK-19504:
-------------------------
Summary: clearCache fails to delete orphan RDDs, especially in
pyspark
Key: SPARK-19504
URL: https://issues.apache.org/jira/browse/SPARK-19504
Project: Spark
Issue Type: Bug
Components: Optimizer
Affects Versions: 2.1.0
Environment: Both pyspark and scala spark. Although scala spark
uncaches some RDD types even if orphan
Reporter: R
Priority: Minor
x=sc.parallelize([1,3,10,9]).cache()
x.count()
x=sc.parallelize([1,3,10,9]).cache()
x.count()
sqlContex.clearCache()
Overwriting x will create an orphan RDD, which cannot be deleted with
clearCache(). This happens in both scala and pyspark.
Similar thing happens for rdds created from dataframe in python
spark.read.csv(....).rdd()
However, in scala clearCache can get rid of some orphan rdd types.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
