[
https://issues.apache.org/jira/browse/HUDI-7153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sivabalan narayanan reassigned HUDI-7153:
-----------------------------------------
Assignee: sivabalan narayanan
> Increasing Kafka minPartitions in Streamer causes corrupted offsets
> -------------------------------------------------------------------
>
> Key: HUDI-7153
> URL: https://issues.apache.org/jira/browse/HUDI-7153
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: Ethan Guo
> Assignee: sivabalan narayanan
> Priority: Blocker
> Fix For: 0.14.1
>
>
> After increasing `hoodie.deltastreamer.source.kafka.minPartitions` for Hudi
> Streamer with Kafka source, at some point, the streamer job fails due to the
> corrupted offset (see below), though the topic is not damaged. Removing the
> config can get over the issue.
> {code:java}
> Caused by: org.apache.spark.SparkException: Job aborted due to stage failure:
> Task 106 in stage 6011.0 failed 4 times, most recent failure: Lost task 106.3
> in stage 6011.0 (TID 429501) (10.151.141.115 executor 28):
> java.lang.IllegalArgumentException: requirement failed: Beginning offset
> -9223372036497672008 is after the ending offset -9223372036497705858 for
> topic *** partition 0. You either provided an invalid fromOffset, or the
> Kafka topic has been damaged
> at scala.Predef$.require(Predef.scala:281)
> at
> org.apache.spark.streaming.kafka010.KafkaRDD.compute(KafkaRDD.scala:186)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
