[
https://issues.apache.org/jira/browse/SPARK-9109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14630467#comment-14630467
]
Sean Owen commented on SPARK-9109:
----------------------------------
unpersist() works correctly actually; it's that the cached RDDs are attached to
other Graphs you create in this process. There are 3 here; think of it as
{code}
val graph1 = GraphGenerators.logNormalGraph(sc, numVertices = 100)
val graph2 = graph1.mapVertices( (id, _) => id.toLong )
val graph3 = graph2.mapEdges( e => e.attr.toLong)
{code}
You'll find 1 of the 2 RDDs left over is from graph1, so call unpersist() on it.
There is a 'leaked' cached edges RDD in here somewhere though, I think, since
I'm still left with one when I try this.
I think there are a number of issues like this in Graphx, where RDDs are cached
but not obviously cleaned up by an unpersist().
Can you try this out, debug, and see if you can pinpoint where the edges RDD
comes from?
> Unpersist a graph object does not work properly
> -----------------------------------------------
>
> Key: SPARK-9109
> URL: https://issues.apache.org/jira/browse/SPARK-9109
> Project: Spark
> Issue Type: Bug
> Components: GraphX
> Affects Versions: 1.3.1, 1.4.0
> Reporter: Tien-Dung LE
> Priority: Minor
>
> Unpersist a graph object does not work properly.
> Here is the code to produce
> {code}
> import org.apache.spark.graphx._
> import org.apache.spark.rdd.RDD
> import org.slf4j.LoggerFactory
> import org.apache.spark.graphx.util.GraphGenerators
> val graph: Graph[Long, Long] =
> GraphGenerators.logNormalGraph(sc, numVertices = 100).mapVertices( (id, _)
> => id.toLong ).mapEdges( e => e.attr.toLong)
>
> graph.cache().numEdges
> graph.unpersist()
> {code}
> There should not be any cached RDDs in storage
> (http://localhost:4040/storage/).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
