that sounds like it. thanks! On Sat, Jan 28, 2017 at 12:49 PM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote:
> Could this be the same issue as the one reported here? > https://issues.apache.org/jira/browse/KAFKA-4547 > > --Vahid > > > > > > From: Koert Kuipers <ko...@tresata.com> > To: users@kafka.apache.org > Date: 01/27/2017 09:34 PM > Subject: possible bug or inconsistency in kafka-clients > > > > hello all, > > i just wanted to point out a potential issue in kafka-clients 0.10.1.1 > > i was using spark-sql-kafka-0-10, which is spark structured streaming > integration for kafka. it depends on kafka-clients 0.10.0.1 but since my > kafka servers are 0.10.1.1 i decided to upgrade kafka-clients to 0.10.1.1 > also. i was not able to read from kafka in spark reliably. the issue > seemed > to be that the kafka consumer got the latest offsets wrong. after > downgrading kafka-clients back to 0.10.0.1 it all worked correctly again. > > did the behavior of KafkaConsumer.seekToEnd change between 0.10.0.1 and > 0.10.1.1? > > for the original discussion see here: > https://www.mail-archive.com/user@spark.apache.org/msg61290.html > > > i think the relevant code in spark is here: > https://github.com/apache/spark/blob/master/external/ > kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/ > KafkaSource.scala#L399 > > > best, > koert > > > > >
