I think now that this is because spark.local.dir is defaulting to /tmp, and since the tasks are not running on the same machine, the file is not found when the second task takes over.
How do you set spark.local.dir appropriately when running on mesos? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Kafka-Mesos-Marathon-strangeness-tp3285p3356.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
