RE: Problems with Local Checkpoints

2015-09-14 Thread Bryan
Akhil, This looks like the issue. I'll update my path to include the (soon to be added) winutils & assoc. DLLs. Thank you, Bryan -Original Message- From: "Akhil Das" Sent: ‎9/‎14/‎2015 6:46 AM To: "Bryan Jeffrey" Cc: "user" Subject: Re: Probl

Re: Problems with Local Checkpoints

2015-09-14 Thread Akhil Das
You need to set your HADOOP_HOME and make sure the winutils.exe is available in the PATH. Here's a discussion around the same issue http://stackoverflow.com/questions/19620642/failed-to-locate-the-winutils-binary-in-the-hadoop-binary-path Also this JIRA https://issues.apache.org/jira/browse/SPARK-

Problems with Local Checkpoints

2015-09-09 Thread Bryan Jeffrey
Hello. I have some basic code that counts numbers using updateStateByKey. I setup a streaming context with checkpointing as follows: def createStreamingContext(masterName : String, checkpointDirectory : String, timeWindow : Int) : StreamingContext = { val sparkConf = new SparkConf().setAppName