[
https://issues.apache.org/jira/browse/SPARK-22832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-22832:
------------------------------------
Assignee: Apache Spark
> BisectingKMeans unpersist unused datasets
> -----------------------------------------
>
> Key: SPARK-22832
> URL: https://issues.apache.org/jira/browse/SPARK-22832
> Project: Spark
> Issue Type: Improvement
> Components: ML
> Affects Versions: 2.3.0
> Reporter: zhengruifeng
> Assignee: Apache Spark
> Priority: Trivial
>
> {code}
> scala> import org.apache.spark.ml.feature._
> import org.apache.spark.ml.feature._
> scala> import org.apache.spark.ml.linalg.{Vector, Vectors}
> import org.apache.spark.ml.linalg.{Vector, Vectors}
> scala> import org.apache.spark.ml.clustering._
> import org.apache.spark.ml.clustering._
> scala>
> scala> val df =
> spark.read.format("libsvm").load("/Users/zrf/Dev/OpenSource/spark/data/mllib/sample_libsvm_data.txt")
> df: org.apache.spark.sql.DataFrame = [label: double, features: vector]
> scala>
> scala> val bkm = new
> BisectingKMeans().setK(5).setMinDivisibleClusterSize(4).setMaxIter(4).setSeed(123)
> bkm: org.apache.spark.ml.clustering.BisectingKMeans =
> bisecting-kmeans_74183547126a
> scala> bkm.fit(df)
> res0: org.apache.spark.ml.clustering.BisectingKMeansModel =
> bisecting-kmeans_74183547126a
> scala> sc.getPersistentRDDs
> res1: scala.collection.Map[Int,org.apache.spark.rdd.RDD[_]] = Map(12 ->
> MapPartitionsRDD[12] at map at BisectingKMeans.scala:151, 54 ->
> MapPartitionsRDD[54] at keys at BisectingKMeans.scala:202)
> {code}
> The {{norms}} and the last {{indices}} but one are not unpersisted after
> training.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
