[
https://issues.apache.org/jira/browse/SPARK-9109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14630941#comment-14630941
]
Tien-Dung LE commented on SPARK-9109:
-------------------------------------
Thanks [~sowen]. Indeed, there is still a cached edges RDD. I think this RDD is
left from the graph construction in GraphImpl.scala, more precisely at
fromEdgeRDD() function.
Here is the latest code follow your suggestion.
{code}
import org.apache.spark.graphx._
import org.apache.spark.rdd.RDD
import org.slf4j.LoggerFactory
import org.apache.spark.graphx.util.GraphGenerators
val graph1 = GraphGenerators.logNormalGraph(sc, numVertices = 100)
val graph2 = graph1.mapVertices( (id, _) => id.toLong )
val graph3 = graph2.mapEdges( e => e.attr.toLong)
graph3.cache().numEdges
graph3.unpersist()
graph2.unpersist()
graph1.unpersist()
graph2.unpersist()
graph3.unpersist()
sc.getPersistentRDDs.foreach( r => println( r._2.toString))
/*
GraphImpl.scala
private def fromEdgeRDD[VD: ClassTag, ED: ClassTag](
edges: EdgeRDDImpl[ED, VD],
defaultVertexAttr: VD,
edgeStorageLevel: StorageLevel,
vertexStorageLevel: StorageLevel): GraphImpl[VD, ED] = {
val edgesCached = edges.withTargetStorageLevel(edgeStorageLevel).cache()
val vertices = VertexRDD.fromEdges(edgesCached,
edgesCached.partitions.size, defaultVertexAttr)
.withTargetStorageLevel(vertexStorageLevel)
fromExistingRDDs(vertices, edgesCached)
}
*/
{code}
> Unpersist a graph object does not work properly
> -----------------------------------------------
>
> Key: SPARK-9109
> URL: https://issues.apache.org/jira/browse/SPARK-9109
> Project: Spark
> Issue Type: Bug
> Components: GraphX
> Affects Versions: 1.3.1, 1.4.0
> Reporter: Tien-Dung LE
> Priority: Minor
>
> Unpersist a graph object does not work properly.
> Here is the code to produce
> {code}
> import org.apache.spark.graphx._
> import org.apache.spark.rdd.RDD
> import org.slf4j.LoggerFactory
> import org.apache.spark.graphx.util.GraphGenerators
> val graph: Graph[Long, Long] =
> GraphGenerators.logNormalGraph(sc, numVertices = 100).mapVertices( (id, _)
> => id.toLong ).mapEdges( e => e.attr.toLong)
>
> graph.cache().numEdges
> graph.unpersist()
> {code}
> There should not be any cached RDDs in storage
> (http://localhost:4040/storage/).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
