hudi-bot opened a new issue, #16323:
URL: https://github.com/apache/hudi/issues/16323
After increasing `hoodie.deltastreamer.source.kafka.minPartitions` for Hudi
Streamer with Kafka source, at some point, the streamer job fails due to the
corrupted offset (see below), though the topic is not damaged. Removing the
config can get over the issue.
{code:java}
Caused by: org.apache.spark.SparkException: Job aborted due to stage
failure: Task 106 in stage 6011.0 failed 4 times, most recent failure: Lost
task 106.3 in stage 6011.0 (TID 429501) (10.151.141.115 executor 28):
java.lang.IllegalArgumentException: requirement failed: Beginning offset
-9223372036497672008 is after the ending offset -9223372036497705858 for topic
*** partition 0. You either provided an invalid fromOffset, or the Kafka topic
has been damaged
at scala.Predef$.require(Predef.scala:281)
at
org.apache.spark.streaming.kafka010.KafkaRDD.compute(KafkaRDD.scala:186)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373) {code}
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-7153
- Type: Bug
- Fix version(s):
- 1.1.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
