Spark streaming - failed recovery from checkpoint

Hi guys, (apologize for the huge font, reposting), I’ve encountered some problems with a crashed Spark Streaming job, when restoring from checkpoint. I’m runnning spark 1.5.1 on Yarn (hadoop 2.6) in cluster mode, reading from Kafka with the direct consumer and a few updateStateByKey stateful

Spark streaming - failed recovery from checkpoint

Hi guys, I’ve encountered some problems with a crashed Spark Streaming job, when restoring from checkpoint. I’m runnning spark 1.5.1 on Yarn (hadoop 2.6) in cluster mode, reading from Kafka with the direct consumer and a few updateStateByKey stateful transformations. After investigating, I thi

FW: Spark Streaming scheduler delay VS driver.cores

Apologies for reposting this to the dev list but I’ve had no luck in getting information about spark.driver.cores on the user list. Happy to create a PR with documentation improvements for the spark.driver.cores config setting after I get some more details. Thanks! -adrian From: Adrian Tanase