Is this possibly related to the recent post on https://issues.apache.org/jira/browse/SPARK-18057 ?
On Mon, Apr 16, 2018 at 11:57 AM, ARAVIND SETHURATHNAM < asethurath...@homeaway.com.invalid> wrote: > Hi, > > We have several structured streaming jobs (spark version 2.2.0) consuming > from kafka and writing to s3. They were running fine for a month, since > yesterday few jobs started failing and I see the below exception in the > failed jobs log, > > > > ```Tried to fetch 473151075 but the returned record offset was 473151072``` > ```GScheduler: ResultStage 0 (start at SparkStreamingTask.java:222) failed > in 77.546 s due to Job aborted due to stage failure: Task 86 in stage 0.0 > failed 4 times, most recent failure: Lost task 86.3 in stage 0.0 (TID 96, > ip-10-120-12-52.ec2.internal, executor 11): java.lang.IllegalStateException: > Tried to fetch 473151075 but the returned record offset was 473151072 > at org.apache.spark.sql.kafka010.CachedKafkaConsumer.fetchData( > CachedKafkaConsumer.scala:234) > at org.apache.spark.sql.kafka010.CachedKafkaConsumer.get( > CachedKafkaConsumer.scala:106) > at org.apache.spark.sql.kafka010.KafkaSourceRDD$$anon$1. > getNext(KafkaSourceRDD.scala:158) > at org.apache.spark.sql.kafka010.KafkaSourceRDD$$anon$1. > getNext(KafkaSourceRDD.scala:149) > at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73) > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408) > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408) > at org.apache.spark.sql.catalyst.expressions.GeneratedClass$ > GeneratedIterator.processNext(Unknown Source) > at org.apache.spark.sql.execution.BufferedRowIterator. > hasNext(BufferedRowIterator.java:43) > at org.apache.spark.sql.execution.WholeStageCodegenExec$$ > anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:395) > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408) > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408) > at scala.collection.convert.Wrappers$IteratorWrapper. > hasNext(Wrappers.scala:30) > ` > > > > can someone provide some direction what could be causing this all of a > sudden when consuming from those topics? > > > > regards > > [image: https://ssl.gstatic.com/ui/v1/icons/mail/images/cleardot.gif] > > Aravind > > >
