I put up https://issues.apache.org/jira/browse/SPARK-10320. Let's continue this conversation there.
From: c...@koeninger.org At: Aug 27 2015 10:30:12 To: Sudarshan Kadambi (BLOOMBERG/ 731 LEX) Cc: user@spark.apache.org Subject: Re: Adding Kafka topics to a running streaming context If you all can say a little more about what your requirements are, maybe we can get a jira together. I think the easiest way to deal with this currently is to start a new job before stopping the old one, which should prevent latency problems. On Thu, Aug 27, 2015 at 9:24 AM, Sudarshan Kadambi (BLOOMBERG/ 731 LEX) <skada...@bloomberg.net> wrote: This is something we have been needing for a while too. We are restarting the streaming context to handle new topic subscriptions & unsubscriptions which affects latency of update handling. I think this is something that needs to be addressed in core Spark Streaming (I can't think of any fundamental limitations that prevent this, perhaps nobody has just expressed interest in this feature so far?). From: yael.aharo...@gmail.com At: Aug 27 2015 10:19:33 To: user@spark.apache.org Subject: Re:Adding Kafka topics to a running streaming context Hello, My streaming application needs to allow consuming new Kafka topics at arbitrary times. I know I can stop and start the streaming context when I need to introduce a new stream, but that seems quite disruptive. I am wondering if other people have this situation and if there is a more elegant solution? thanks, Yael
