Re: GraphX can not unpersist edges of old graph?

[Ankur Dave]

We didn't provide an unpersist API for Graph because the internal
dependency structure of a graph can make it hard to unpersist correctly in
a way that avoids recomputation. However, you can directly unpersist a
graph's vertices and edges RDDs using graph.vertices.unpersist() and
graph.edges.unpersist().

By the way, the memory leak bug with Pregel (SPARK-2025
) is fixed in master.

Ankur 

GraphX can not unpersist edges of old graph?

[余根茂(余根茂)]

Hi All,

 

I know the benefit of RDD caching, but abuse of using cache may cause memory
leak. In graphx, we can cache a graph by using graph.cache(), and many
transformation of graph create and cache new edges, like partitionBy() and
subgraph(). However, I can not find an interface to unpersist edges. I
wonder the purpose of design like this. It can indeed improve performance.
But, it may lead to memory leak if not unpersist edges in some cases. For
example, in Spark-1.0,  the pregel can not unpersist edges of old graph
effectively, and leads to memory leak.

 

 

Much Thanks!