- are you using different group ids for the different streams? - are you manually committing offsets? - what are the values of your kafka-related settings?
On Fri, Nov 4, 2016 at 12:20 PM, vonnagy <i...@vadio.com> wrote: > I am getting the issues using Spark 2.0.1 and Kafka 0.10. I have two jobs, > one that uses a Kafka stream and one that uses just the KafkaRDD. > > With the KafkaRDD, I continually get the "Failed to get records .. after > polling". I have adjusted the polling with > `spark.streaming.kafka.consumer.poll.ms` and the size of records with > Kafka's `max.poll.records`. Even when it gets records it is extremely slow. > > When working with multiple KafkaRDDs in parallel I get the dreaded > `ConcurrentModificationException`. The Spark logic is supposed to use a > CachedKafkaConsumer based on the topic and partition. This is supposed to > guarantee thread safety, but I continually get this error along with the > polling timeout. > > Has anyone else tried to use Spark 2 with Kafka 0.10 and had any success. At > this point it is completely useless in my experience. With Spark 1.6 and > Kafka 0.8.x, I never had these problems. > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Instability-issues-with-Spark-2-0-1-and-Kafka-0-10-tp28017.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
