Hi, If you disable tracker-side WAL, you unset a checkpoint dir by using streamingContext.checkpoint(). http://spark.apache.org/docs/latest/streaming-programming-guide.html#how-to-configure-checkpointing
// maropu On Tue, Nov 29, 2016 at 9:04 AM, Tim Harsch <thar...@cray.com> wrote: > Hi all, > > I set `spark.streaming.receiver.writeAheadLog.enable=false` and my > history server confirms the property has been set. Yet, I continue to see > the error: > > > > 16/11/28 15:47:04 ERROR util.FileBasedWriteAheadLog_ReceivedBlockTracker: > Failed to write to write ahead log after 3 failures > > > > I surmised from the spark users thread Spark Streaming Data loss on > failure to write BlockAdditionEvent failure to WAL ( > https://www.mail-archive.com/user@spark.apache.org/msg58952.html) that > HDFS append support must be enabled for WAL to work with HDFS. My > installation does not enable this HDFS feature, so I would like to disable > WAL in Spark. > > > > Thanks, > > Tim > > > -- --- Takeshi Yamamuro
