Nitin, I am getting the similar issues using Spark 2.0.1 and Kafka 0.10. I have to jobs, one that uses a Kafka stream and one that uses just the KafkaRDD.
With the KafkaRDD, I continually get the "Failed to get records". I have adjusted the polling with `spark.streaming.kafka.consumer.poll.ms` and the size of records with Kafka's `max.poll.records`. Even when it gets records it is extremely slow. When working with multiple KafkaRDDs in parallel I get the dreaded `ConcurrentModificationException`. The Spark logic is supposed to use a CachedKafkaConsumer based on the topic and partition. This is supposed to guarantee thread safety, but I continually get this error along with the polling timeout. Has anyone else tried to use Spark 2 with Kafka 0.10 and had any success. At this point it is completely useless in my experience. With Spark 1.6 and Kafka 0.8.x, I never had these problems. -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Continuous-warning-while-consuming-using-new-kafka-spark010-API-tp18987p19736.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
