Are you using receiver-based or direct stream? Are you doing 1 stream per topic, or 1 stream for all topics?
If you're using the direct stream, the actual topics and offset ranges should be visible in the logs, so you should be able to see more detail about what's happening (e.g. all topics are still being processed but offsets are significantly behind, vs only certain topics being processed but keeping up with latest offsets) On Mon, Jan 23, 2017 at 3:14 PM, hakanilter <hakanil...@gmail.com> wrote: > Hi everyone, > > I have a spark (1.6.0-cdh5.7.1) streaming job which receives data from > multiple kafka topics. After starting the job, everything works fine first > (like 700 req/sec) but after a while (couples of days or a week) it starts > processing only some part of the data (like 350 req/sec). When I check the > kafka topics, I can see that there are still 700 req/sec coming to the > topics. I don't see any errors, exceptions or any other problem. The job > works fine when I start the same code with just single kafka topic. > > Do you have any idea or a clue to understand the problem? > > Thanks. > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Spark-streaming-multiple-kafka-topic-doesn-t-work-at-least-once-tp28334.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
