[
https://issues.apache.org/jira/browse/SPARK-8480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15827515#comment-15827515
]
Kaushal Prajapati commented on SPARK-8480:
------------------------------------------
For example, Using SparkContext we can get all cached Rdds
{code:title=Code|borderStyle=solid}
scala> val rdd = sc.range(1,1000)
scala> rdd.setName("myRdd")
scala> rdd.cache
scala> rdd.count
scala> sc.getPersistentRDDs.foreach(println)
(9,kaushal MapPartitionsRDD[9] at rdd at <console>:27)
(11,myRdd MapPartitionsRDD[11] at range at <console>:27)
sc.getPersistentRDDs.filter(_._2.name == "myRdd").foreach(_._2.unpersist())
scala> sc.getPersistentRDDs.foreach(println)
(9,kaushal MapPartitionsRDD[9] at rdd at <console>:27)
{code}
And we can unpersist any Rdd with valid name.
Likewise same in DataSet, if we will able to list all cached DataSets with
corresponding names then it will be good option to unpersist any DataSet using
particular name.
> Add setName for Dataframe
> -------------------------
>
> Key: SPARK-8480
> URL: https://issues.apache.org/jira/browse/SPARK-8480
> Project: Spark
> Issue Type: Wish
> Components: SQL
> Affects Versions: 1.4.0
> Reporter: Peter Rudenko
> Priority: Minor
>
> Rdd has a method setName, so in spark UI, it's more easily to understand
> what's this cache for. E.g. ("data for LogisticRegression model", etc.).
> Would be nice to have the same method for Dataframe, since it displays a
> logical schema, in cache page, which could be quite big.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
