[
https://issues.apache.org/jira/browse/KAFKA-15608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18112508#comment-18112508
]
hyungmin OH commented on KAFKA-15608:
-------------------------------------
Re-reported as KAFKA-20961 (JBOD log-directory move on a broker that is already
leader; same mechanism). Fix with a deterministic reproduction on trunk:
https://github.com/apache/kafka/pull/23205
> Uncopied leader epoch cache cause repeated OffsetOutOfRangeException
> --------------------------------------------------------------------
>
> Key: KAFKA-15608
> URL: https://issues.apache.org/jira/browse/KAFKA-15608
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 3.1.0
> Reporter: Drawxy
> Priority: Major
>
> Recently, I encountered a issue that there was always 1 partition having only
> 1 ISR (no produce traffic on this topic). The bug is related to altering log
> dir. When replacing current log with future log, broker doesn't copy the
> leader epoch checkpoint cache, which records the current leader epoch and log
> start offset. The cache for each partition is updated only when appending new
> messages or becoming leader. If there is no traffic and the replica is
> already the leader, the cache will not be updated any more. However, the
> partition leader will fetch its leader epoch from the cache and compare with
> the leader epoch sent by follower when handling fetch request. If the former
> one is missed or less than the latter one, the leader will interrupt the
> process and return an OffsetOutOfRangeException to follower. The follower
> might be out of sync over time.
> Take the following case as an example, all the key points are listed in
> chronological order:
> # Reassigner submitted a partition reassignment for partition foo-1
> {quote}
> {{ {}}
> {{ "topic": "foo",}}
> {{ "partition": 1,}}
> {{ "replicas": [}}
> {{ 5002,}}
> {{ 3003,}}
> {{ 4001}}
> {{ ],}}
> {{ "logDirs": [}}
> {{ "\\data\\kafka-logs-1",}}
> {{ "any",}}
> {{ "any"}}
> {{ ]}}
> {{ } }}
>
> {quote}
> # Reassignment completed immediately due to there is no traffic on this
> topic.
> # Controller sent LeaderAndISR requests to all the replicas.
> # Newly added replica 5002 became the new leader and the current log updated
> the leader epoch offset cache. Replica 5002 successfully handled the
> LeaderAndISR request.
> # Altering log dir completed and the newly updated current log didn't have
> leader epoch offset information.
> # Replica 5002 handled fetch requests (include fetch offset and current
> leader epoch) from followers and returned OffsetOutOfRangeException due to
> leader epoch offset cache hadn't been updated. So, the replica 5002 couldn't
> update the fetch state for each follower and reported ISRShrink later. The
> followers 3003 and 4001 would repeatedly print the following log:
> {quote}{{WARN [ReplicaFetcher replicaId=4001, leaderId=5002, fetcherId=2]
> Reset fetch offset for partition foo-1 from 231196 to current leader's start
> offset 231196 (kafka.server.ReplicaFetcherThread)}}
> {{INFO [ReplicaFetcher replicaId=4001, leaderId=5002, fetcherId=2] Current
> offset 231196 for partition foo-1 is out of range, which typically implies a
> leader change. Reset fetch offset to 231196
> (kafka.server.ReplicaFetcherThread)}}{quote}
> This issue arises only when all the three conditions are met:
> # No produce traffic on the partition.
> # Newly added replica become new leader.
> # LeaderAndISR request is handled successfully before altering log dir
> completed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
