abhishekrb19 commented on issue #14344: URL: https://github.com/apache/druid/issues/14344#issuecomment-3045783328
Like others, we encountered this issue on a cluster running Druid 32.0.1. After performing a manual hard reset of the supervisor manually, we saw this issue surface eventually for a few partitions. We have auto-reset disabled `resetOffsetAutomatically: false` and `useEarliestOffset: true`. The requested offset is less than the earliest available offset, which makes this an irrecoverable state. However, the task simply logs a warning, retries, and cycles through without consuming any messages. We only caught the issue due to a noticeable lag build-up. Looks like the code assumes that this is a legitimate situation in all cases and returns an empty set of records. This may be the case for when auto-reset is enabled or when a topic partition isn't available yet, but there are other scenarios, like the one above, where the requested offset is no longer available and valid - https://github.com/apache/druid/blob/master/extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/KafkaIndexTaskRunner.java#L99 ``` org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner - OffsetOutOfRangeException with message [Fetch position FetchPosition{offset=2380443635, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[broker-config:9093 (id: 0 rack: us-west-2a)], epoch=44}} is out of range for partition xyz-14] ``` >look at the exception handler of OffsetOutOfRangeException, I can't remember why it waits to retry if auto-reset is not enabled. under such case, the retry seems useless, there's no way to recover from it automatically. @FrankChen021, agreed. For the case when `resetOffsetAutomatically: false`, we should just fail the task with a useful message so operators can take a corrective action to reset the offsets for the problematic partitions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
