[
https://issues.apache.org/jira/browse/HUDI-7153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ethan Guo updated HUDI-7153:
----------------------------
Description:
After increasing `hoodie.deltastreamer.source.kafka.minPartitions` for Hudi
Streamer with Kafka source, at some point, the streamer job fails due to the
corrupted offset (see below), though the topic is not damaged. Removing the
config can get over the issue.
{code:java}
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure:
Task 106 in stage 6011.0 failed 4 times, most recent failure: Lost task 106.3
in stage 6011.0 (TID 429501) (10.151.141.115 executor 28):
java.lang.IllegalArgumentException: requirement failed: Beginning offset
-9223372036497672008 is after the ending offset -9223372036497705858 for topic
*** partition 0. You either provided an invalid fromOffset, or the Kafka topic
has been damaged
at scala.Predef$.require(Predef.scala:281)
at org.apache.spark.streaming.kafka010.KafkaRDD.compute(KafkaRDD.scala:186)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373) {code}
> Increasing Kafka minPartitions in Streamer causes corrupted offsets
> -------------------------------------------------------------------
>
> Key: HUDI-7153
> URL: https://issues.apache.org/jira/browse/HUDI-7153
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: Ethan Guo
> Priority: Major
> Fix For: 0.14.1
>
>
> After increasing `hoodie.deltastreamer.source.kafka.minPartitions` for Hudi
> Streamer with Kafka source, at some point, the streamer job fails due to the
> corrupted offset (see below), though the topic is not damaged. Removing the
> config can get over the issue.
> {code:java}
> Caused by: org.apache.spark.SparkException: Job aborted due to stage failure:
> Task 106 in stage 6011.0 failed 4 times, most recent failure: Lost task 106.3
> in stage 6011.0 (TID 429501) (10.151.141.115 executor 28):
> java.lang.IllegalArgumentException: requirement failed: Beginning offset
> -9223372036497672008 is after the ending offset -9223372036497705858 for
> topic *** partition 0. You either provided an invalid fromOffset, or the
> Kafka topic has been damaged
> at scala.Predef$.require(Predef.scala:281)
> at
> org.apache.spark.streaming.kafka010.KafkaRDD.compute(KafkaRDD.scala:186)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
