Sergey Zhemzhitsky created SPARK-22184: ------------------------------------------
Summary: GraphX fails in case of insufficient memory and checkpoints enabled Key: SPARK-22184 URL: https://issues.apache.org/jira/browse/SPARK-22184 Project: Spark Issue Type: Bug Components: GraphX Affects Versions: 2.2.0 Environment: spark 2.2.0 scala 2.11 Reporter: Sergey Zhemzhitsky Fix For: 2.2.1 GraphX fails with FileNotFoundException in case of insufficient memory when checkpoints are enabled. Here is the stacktrace {code} Job aborted due to stage failure: Task creation failed: java.io.FileNotFoundException: File file:/tmp/spark-90119695-a126-47b5-b047-d656fee10c17/9b16e2a9-6c80-45eb-8736-bbb6eb840146/rdd-28/part-00000 does not exist java.io.FileNotFoundException: File file:/tmp/spark-90119695-a126-47b5-b047-d656fee10c17/9b16e2a9-6c80-45eb-8736-bbb6eb840146/rdd-28/part-00000 does not exist at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:539) at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:752) at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:529) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:409) at org.apache.spark.rdd.ReliableCheckpointRDD.getPreferredLocations(ReliableCheckpointRDD.scala:89) at org.apache.spark.rdd.RDD$$anonfun$preferredLocations$1.apply(RDD.scala:274) at org.apache.spark.rdd.RDD$$anonfun$preferredLocations$1.apply(RDD.scala:274) at scala.Option.map(Option.scala:146) at org.apache.spark.rdd.RDD.preferredLocations(RDD.scala:274) at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal(DAGScheduler.scala:1697) ... {code} As GraphX uses cached RDDs intensively, the issue is only reproducible when previously cached and checkpointed Vertex and Edge RDDs are evicted from memory and forced to be read from disk. For testing purposes the following parameters may be set to emulate low memory environment {code} val sparkConf = new SparkConf() .set("spark.graphx.pregel.checkpointInterval", "2") // set testing memory to evict cached RDDs from it and force // reading checkpointed RDDs from disk .set("spark.testing.reservedMemory", "128") .set("spark.testing.memory", "256") {code} This issue also includes SPARK-22150 and cannot be fixed until SPARK-22150 is fixed too. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org