Hi all, I have a question about long running streaming applications and workers that act as consumers.
Specifically my program runs on a spark standalone cluster with a small number of workers, acting as kafka consumers using spark streaming. What I noticed was that in a long running application, if one of the workers dies for some reason and then a new worker registers to replace it, we have effectively lost that worker as a consumer. When the driver first runs, I create a configured amount of KafkaInputDStream instances, in my case, the same number as the number of workers in the cluster, and spark distributes these among the workers, so each one of my workers consumes from Kafka. I then unify the streams to a single stream using SparkStreamingContext union. This code never runs again though, and there is no code that monitors that we have X number of consumers at all time. So when a worker dies, we effectively lose a consumer, and never create a new one, then the lag in Kafka starts growing. Does anybody have a solution / ideas regarding this issue? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Long-running-streaming-application-worker-death-tp23997.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
