Github user bxshi commented on the pull request:
https://github.com/apache/spark/pull/1177#issuecomment-76080970
It's not as simple as I thought...
Here is my reply on JIRA about this PR
I edited my original comment to add the updates, but I do not know if you
can get them via email. So I resubmit it again. Hope that won't bother you.
Ankur Dave
Hi Ankur Dave, I changed my pull request. But there is another exception,
ShippableVertexPartition is not serializable. So I serialized it, but there is
another exception org.apache.spark.graphx.impl.RoutingTablePartition is not
serializable. Then I serialized it again, but on iteration 2 there will be an
exception: org.apache.spark.graphx.impl.ShippableVertexPartition cannot be cast
to scala.Tuple2
The code I'm using are:
```
val conf = new SparkConf().setAppName("HDTM")
.setMaster("local[4]")
val sc = new SparkContext(conf)
sc.setCheckpointDir("./checkpoint")
val v = sc.parallelize(Seq[(VertexId, Long)]((0L, 0L), (1L, 1L), (2L, 2L)))
val e = sc.parallelize(Seq[Edge[Long]](Edge(0L, 1L, 0L), Edge(1L, 2L, 1L),
Edge(2L, 0L, 2L)))
var g = Graph(v, e)
val vertexIds = Seq(0L, 1L, 2L)
var prevG: Graph[VertexId, Long] = null
for (i <- 1 to 2000) {
vertexIds.toStream.foreach(id =>
{ prevG = g g = Graph(g.vertices, g.edges) g.vertices.cache()
g.edges.cache() prevG.unpersistVertices(blocking = false)
prevG.edges.unpersist(blocking = false) }
)
g.vertices.checkpoint()
g.edges.checkpoint()
g.edges.count()
g.vertices.count()
println(s"$
{g.vertices.isCheckpointed}
$
{g.edges.isCheckpointed}
")
println(" iter " + i + " finished")
}
println(g.vertices.collect().mkString(" "))
println(g.edges.collect().mkString(" "))
```
Am I on the right track? Or Should there be another way to change it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
