I have disabled checkpointing in my Spark Streaming application with:
ssc.checkpoint(null)
This application does not need checkpointing, and there is no operation
that I'm doing (only filter operations, and, importantly, NO WINDOWING)
that should need checkpointing. And yet I am getting this:
2013-10-29 14:47:22 ERROR LocalScheduler - Exception in task 3
java.lang.Exception: Could not compute split, block input-0-1383076026200
not found
at spark.rdd.BlockRDD.compute(BlockRDD.scala:32)
at spark.RDD.computeOrReadCheckpoint(RDD.scala:207)
at spark.RDD.iterator(RDD.scala:196)
at spark.rdd.MappedRDD.compute(MappedRDD.scala:12)
at spark.RDD.computeOrReadCheckpoint(RDD.scala:207)
at spark.RDD.iterator(RDD.scala:196)
at spark.rdd.FilteredRDD.compute(FilteredRDD.scala:15)
at spark.RDD.computeOrReadCheckpoint(RDD.scala:207)
at spark.RDD.iterator(RDD.scala:196)
at spark.rdd.MappedRDD.compute(MappedRDD.scala:12)
at spark.RDD.computeOrReadCheckpoint(RDD.scala:207)
at spark.RDD.iterator(RDD.scala:196)
at spark.rdd.UnionPartition.iterator(UnionRDD.scala:12)
at spark.rdd.UnionRDD.compute(UnionRDD.scala:52)
at spark.RDD.computeOrReadCheckpoint(RDD.scala:207)
at spark.RDD.iterator(RDD.scala:196)
at spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:19)
at spark.RDD.computeOrReadCheckpoint(RDD.scala:207)
at spark.RDD.iterator(RDD.scala:196)
at spark.scheduler.ShuffleMapTask.run(ShuffleMapTask.scala:127)
at spark.scheduler.ShuffleMapTask.run(ShuffleMapTask.scala:75)
at
spark.scheduler.local.LocalScheduler.runTask$1(LocalScheduler.scala:76)
at
spark.scheduler.local.LocalScheduler$$anon$1.run(LocalScheduler.scala:49)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Why might I be getting this exception?
Thanks,
Craig Vanderborgh