Re: Streaming | Partition count mismatch exception while saving data in RDD

2014-12-16 Thread Aniket Bhatnagar
It turns out that this happens when checkpoint is set to a local directory path. I have opened a JIRA SPARK-4862 for Spark streaming to output better error message. Thanks, Aniket On Tue Dec 16 2014 at 20:08:13 Aniket Bhatnagar wrote: > I am using spark 1.1.0 running a streaming job that uses u

Streaming | Partition count mismatch exception while saving data in RDD

2014-12-16 Thread Aniket Bhatnagar
I am using spark 1.1.0 running a streaming job that uses updateStateByKey and then (after a bunch of maps/flatMaps) does a foreachRDD to save data in each RDD by making HTTP calls. The issue is that each time I attempt to save the RDD (using foreach on RDD), it gives me the following exception: or