Re: Adding Kafka topics to a running streaming context

2015-08-27 Thread Sudarshan Kadambi (BLOOMBERG/ 731 LEX)
I put up https://issues.apache.org/jira/browse/SPARK-10320. Let's continue this conversation there. From: c...@koeninger.org At: Aug 27 2015 10:30:12 To: Sudarshan Kadambi (BLOOMBERG/ 731 LEX) Cc: user@spark.apache.org Subject: Re: Adding Kafka topics to a running streaming context If yo

Re: Adding Kafka topics to a running streaming context

2015-08-27 Thread Cody Koeninger
If you all can say a little more about what your requirements are, maybe we can get a jira together. I think the easiest way to deal with this currently is to start a new job before stopping the old one, which should prevent latency problems. On Thu, Aug 27, 2015 at 9:24 AM, Sudarshan Kadambi (BL

Adding Kafka topics to a running streaming context

2015-08-27 Thread yael aharon
Hello, My streaming application needs to allow consuming new Kafka topics at arbitrary times. I know I can stop and start the streaming context when I need to introduce a new stream, but that seems quite disruptive. I am wondering if other people have this situation and if there is a more elegant s