This is a known issue introduced in Spark 1.4.1 and 1.5.0 and will be fixed in Spark 1.5.1. In the mean time, you could prototype in Spark 1.4.0 and wait for Spark 1.5.1/1.4.2 to come out. You could also download the source code and compile the Spark master branch. https://issues.apache.org/jira/browse/SPARK-10071
On Tue, Sep 8, 2015 at 6:46 AM, Bryan Jeffrey <bryan.jeff...@gmail.com> wrote: > Hello. We're getting started with Spark Streaming. We're working to build > some unit/acceptance testing around functions that consume DStreams. The > current method for creating DStreams is to populate the data by creating an > InputDStream: > > val input = Array(TestDataFactory.CreateEvent(123 notFoundData)) > val queue = > scala.collection.mutable.Queue(ssc.sparkContext.parallelize(input)) > val events: InputDStream[MyEvent] = ssc.queueStream(queue) > > The 'events' InputDStream can then be fed into functions. However, the > stream does not allow checkpointing. This means that we're unable to use > this to feed methods/classes that execute stateful actions like > 'updateStateByKey'. > > Does anyone have a simple, contained method to create DStreams that allow > for checkpointing? I looked at the Spark unit test framework, but that > seems to require access to a bunch of spark internals (requiring that > you're within the spark package, etc.) > >
