Hi,
I'm using Kafka 2.5.1 broker and Kafka Connect Confluent image 7.1.1. We
are using a sink connector to read from Kafka.
We occasionally see Fetch Position OutOfRange error like this
[2024-07-19 00:54:59,456] INFO [Consumer
> clientId=connector-consumer-CPSSectorRouterTestEventSinkConnector-1,
> groupId=connect-CPSSectorRouterTestEventSinkConnector] Fetch position
> FetchPosition{offset=897705, offsetEpoch=Optional[0],
> currentLeader=LeaderAndEpoch{leader=Optional[10.26.53.192:9092 (id: 2
> rack: null)], epoch=0}} is out of range for partition
> producer-perf-test-v6-1, resetting offset
> (org.apache.kafka.clients.consumer.internals.Fetcher)
> [2024-07-19 00:54:59,458] INFO [Consumer
> clientId=connector-consumer-CPSSectorRouterTestEventSinkConnector-1,
> groupId=connect-CPSSectorRouterTestEventSinkConnector] Resetting offset for
> partition producer-perf-test-v6-1 to position FetchPosition{offset=598398,
> offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
> 10.26.53.192:9092 (id: 2 rack: null)], epoch=0}}.
> (org.apache.kafka.clients.consumer.internals.SubscriptionState)
On the Connect side we use offset.commit.reset=earliest as moving it to
latest might cause data loss.
On the broker side, we are using the default configurations for
log.segment.bytes and log.retention.bytes but when we see the Fetch
Position OutOfRange, we see an error on the broker side like below.
DEBUG Completed request:RequestHeader(apiKey=FETCH, apiVersion=11,
> clientId=connector-consumer-CPSSectorRouterTestEventSinkConnector-2,
> correlationId=1913054) --
> {replica_id=-1,max_wait_time=500,min_bytes=1,max_bytes=52428800,isolation_level=0,session_id=1832635140,session_epoch=1911234,topics=[{topic=producer-perf-test-v6,partitions=[{partition=2,current_leader_epoch=5,fetch_offset=20200612,log_start_offset=-1,partition_max_bytes=1048576}]}],forgotten_topics_data=[],rack_id=},response:{throttle_time_ms=0,error_code=0,session_id=1832635140,
> responses=[{topic=producer-perf-test-v6,partition_responses=[{partition_header={partition=2,error_code=1,high_watermark=-1,last_stable_offset=-1,log_start_offset=-1,aborted_transactions=null,preferred_read_replica=-1},record_set=MemoryRecords(size=0,
> buffer=java.nio.HeapByteBuffer[pos=0 lim=0 cap=0])}]}]} from connection
> 10.26.53.192:9092-10.26.48.167:51378-7;totalTime:0.748,requestQueueTime:0.145,localTime:0.576,remoteTime:0.0,throttleTime:0.353,responseQueueTime:0.012,sendTime:0.023,securityProtocol:PLAINTEXT,principal:User:ANONYMOUS,listener:PLAINTEXT
> (kafka.request.logger)
I see that we roll segments once every 5-6 mins. But the Fetch Position
OutOfRange doesn't necessarily match with the segment roll over.
I thought this was caused due
https://issues.apache.org/jira/browse/KAFKA-9543 but this should have been
fixed in 2.5.1
Was this bug fixed in 2.5.1? And what else could be the reason for this?
Appreciate any pointers
Thanks
Kishore
