I am getting the issues using Spark 2.0.1 and Kafka 0.10. I have two jobs, one that uses a Kafka stream and one that uses just the KafkaRDD.
With the KafkaRDD, I continually get the "Failed to get records .. after polling". I have adjusted the polling with `spark.streaming.kafka.consumer.poll.ms` and the size of records with Kafka's `max.poll.records`. Even when it gets records it is extremely slow. When working with multiple KafkaRDDs in parallel I get the dreaded `ConcurrentModificationException`. The Spark logic is supposed to use a CachedKafkaConsumer based on the topic and partition. This is supposed to guarantee thread safety, but I continually get this error along with the polling timeout. Has anyone else tried to use Spark 2 with Kafka 0.10 and had any success. At this point it is completely useless in my experience. With Spark 1.6 and Kafka 0.8.x, I never had these problems. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Instability-issues-with-Spark-2-0-1-and-Kafka-0-10-tp28017.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
