Hey Craig, Maybe I'm missing something, but I don't see any evidence that your code is doing checkpointing.
This error is caused by your job looking for an input block that it thinks should exist from the receiver, but it can't find it. What type of input are you using (kafka?). Also, does this happen all the time or is it non-deterministic? - Patrick On Tue, Oct 29, 2013 at 1:51 PM, Craig Vanderborgh <[email protected]> wrote: > I have disabled checkpointing in my Spark Streaming application with: > > ssc.checkpoint(null) > > This application does not need checkpointing, and there is no operation that > I'm doing (only filter operations, and, importantly, NO WINDOWING) that > should need checkpointing. And yet I am getting this: > > 2013-10-29 14:47:22 ERROR LocalScheduler - Exception in task 3 > java.lang.Exception: Could not compute split, block input-0-1383076026200 > not found > at spark.rdd.BlockRDD.compute(BlockRDD.scala:32) > at spark.RDD.computeOrReadCheckpoint(RDD.scala:207) > at spark.RDD.iterator(RDD.scala:196) > at spark.rdd.MappedRDD.compute(MappedRDD.scala:12) > at spark.RDD.computeOrReadCheckpoint(RDD.scala:207) > at spark.RDD.iterator(RDD.scala:196) > at spark.rdd.FilteredRDD.compute(FilteredRDD.scala:15) > at spark.RDD.computeOrReadCheckpoint(RDD.scala:207) > at spark.RDD.iterator(RDD.scala:196) > at spark.rdd.MappedRDD.compute(MappedRDD.scala:12) > at spark.RDD.computeOrReadCheckpoint(RDD.scala:207) > at spark.RDD.iterator(RDD.scala:196) > at spark.rdd.UnionPartition.iterator(UnionRDD.scala:12) > at spark.rdd.UnionRDD.compute(UnionRDD.scala:52) > at spark.RDD.computeOrReadCheckpoint(RDD.scala:207) > at spark.RDD.iterator(RDD.scala:196) > at spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:19) > at spark.RDD.computeOrReadCheckpoint(RDD.scala:207) > at spark.RDD.iterator(RDD.scala:196) > at spark.scheduler.ShuffleMapTask.run(ShuffleMapTask.scala:127) > at spark.scheduler.ShuffleMapTask.run(ShuffleMapTask.scala:75) > at > spark.scheduler.local.LocalScheduler.runTask$1(LocalScheduler.scala:76) > at > spark.scheduler.local.LocalScheduler$$anon$1.run(LocalScheduler.scala:49) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > Why might I be getting this exception? > > Thanks, > Craig Vanderborgh
