GitHub user tien-dungle opened a pull request:
https://github.com/apache/spark/pull/7469
[SPARK-9109] [GraphX] Keep the cached edge in the graph
The change here is to keep the cached RDDs in the graph object so that when
the graph.unpersist() is called these RDDs are correctly unpersisted.
<code>
import org.apache.spark.graphx._
import org.apache.spark.rdd.RDD
import org.slf4j.LoggerFactory
import org.apache.spark.graphx.util.GraphGenerators
// Create an RDD for the vertices
val users: RDD[(VertexId, (String, String))] =
sc.parallelize(Array((3L, ("rxin", "student")), (7L, ("jgonzal",
"postdoc")),
(5L, ("franklin", "prof")), (2L, ("istoica",
"prof"))))
// Create an RDD for edges
val relationships: RDD[Edge[String]] =
sc.parallelize(Array(Edge(3L, 7L, "collab"), Edge(5L, 3L, "advisor"),
Edge(2L, 5L, "colleague"), Edge(5L, 7L, "pi")))
// Define a default user in case there are relationship with missing user
val defaultUser = ("John Doe", "Missing")
// Build the initial Graph
val graph = Graph(users, relationships, defaultUser)
graph.cache().numEdges
graph.unpersist()
sc.getPersistentRDDs.foreach( r => println( r._2.toString))
<code>
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tien-dungle/spark SPARK-9109_Graphx-unpersist
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/7469.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #7469
----
commit 8d879979241050be5918d9630bc5bdec9f62be79
Author: tien-dungle <[email protected]>
Date: 2015-07-17T14:25:25Z
Keep the cached edge in the graph
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
