Xinghao Pan created SPARK-6808: ---------------------------------- Summary: Checkpointing after zipPartitions results in NODE_LOCAL execution Key: SPARK-6808 URL: https://issues.apache.org/jira/browse/SPARK-6808 Project: Spark Issue Type: Bug Components: GraphX, Spark Core Affects Versions: 1.3.0, 1.2.1 Environment: EC2 Ubuntu r3.8xlarge machines Reporter: Xinghao Pan
I'm encountering a weird issue where a simple iterative zipPartition is PROCESS_LOCAL before checkpointing, but turns NODE_LOCAL for all iterations after checkpointing. Here's an example snippet of code: var R : RDD[(Long,Int)] = sc.parallelize((0 until numPartitions), numPartitions) .mapPartitions(_ => new Array[(Long,Int)](10000000).map(i => (0L,0)).toSeq.iterator).cache() sc.setCheckpointDir(checkpointDir) var iteration = 0 while (iteration < 50){ R = R.zipPartitions(R)((x,y) => x).cache() if ((iteration+1) % checkpointIter == 0) R.checkpoint() R.foreachPartition(_ => {}) iteration += 1 } I've also tried to unpersist the old RDDs, and increased spark.locality.wait but nether helps. Strangely, by adding a simple identity map R = R.map(x => x).cache() after the zipPartitions appears to partially mitigate the issue. The problem was originally triggered when I attempted to checkpoint after doing joinVertices in GraphX, but the above example shows that the issue is in Spark core too. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org