[
https://issues.apache.org/jira/browse/SPARK-27636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16833462#comment-16833462
]
shahid commented on SPARK-27636:
--------------------------------
I will raise a PR
> Remove cached RDD blocks after PIC execution
> --------------------------------------------
>
> Key: SPARK-27636
> URL: https://issues.apache.org/jira/browse/SPARK-27636
> Project: Spark
> Issue Type: Improvement
> Components: MLlib
> Affects Versions: 2.3.3, 2.4.2, 3.0.0
> Reporter: shahid
> Priority: Major
>
> Test steps to reproduce:
> 1) bin/spark-shell
> val dataset = spark.createDataFrame(Seq(
> (0L, 1L, 1.0),
> (1L,2L,1.0),
> (3L, 4L,1.0),
> (4L,0L,0.1))).toDF("src", "dst", "weight")
> val model = new PowerIterationClustering().
> setMaxIter(10).
> setInitMode("degree").
> setWeightCol("weight")
> val prediction = model.assignClusters(dataset).select("id", "cluster")
> 2) Open storage tab of the UI. we can see many RDD block cached, even after
> running the PIC.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
