Could this be the same issue as the one reported here? https://issues.apache.org/jira/browse/KAFKA-4547
--Vahid From: Koert Kuipers <ko...@tresata.com> To: users@kafka.apache.org Date: 01/27/2017 09:34 PM Subject: possible bug or inconsistency in kafka-clients hello all, i just wanted to point out a potential issue in kafka-clients 0.10.1.1 i was using spark-sql-kafka-0-10, which is spark structured streaming integration for kafka. it depends on kafka-clients 0.10.0.1 but since my kafka servers are 0.10.1.1 i decided to upgrade kafka-clients to 0.10.1.1 also. i was not able to read from kafka in spark reliably. the issue seemed to be that the kafka consumer got the latest offsets wrong. after downgrading kafka-clients back to 0.10.0.1 it all worked correctly again. did the behavior of KafkaConsumer.seekToEnd change between 0.10.0.1 and 0.10.1.1? for the original discussion see here: https://www.mail-archive.com/user@spark.apache.org/msg61290.html i think the relevant code in spark is here: https://github.com/apache/spark/blob/master/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSource.scala#L399 best, koert
