Hello, I was interested in creating a StreamingContext textFileStream based job, which runs for long durations, and can also recover from prolonged driver failure... It seems like StreamingContext checkpointing is mainly used for the case when the driver dies during the processing of an RDD, and to recover that one RDD, but my question specifically relates to whether there is a way to also recover which files were missed between the timeframe of the driver dying and being started back up (whether manually or automatically).
Any assistance/suggestions with this one would be greatly appreciated! Thanks, Mark. -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/StreamingContext-textFileStream-question-tp10742.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
