[
https://issues.apache.org/jira/browse/KAFKA-20961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18112507#comment-18112507
]
hyungmin OH commented on KAFKA-20961:
-------------------------------------
This looks like the same defect as KAFKA-15608, still present on trunk.
LogManager's swap of the current log with the future log carries the high
watermark over but not the leader epoch cache, and the future log only learns
epochs from the record batches copied into it. An epoch with no records behind
it - one is assigned on every leader election - is therefore lost. The leader
can then no longer resolve the follower's last fetched epoch and throws
OffsetOutOfRangeException; the follower's recovery path finds nothing to
truncate on an idle partition, so it repeats the identical fetch, which matches
the ~12ms loop reported here.
PR #14553 proposed a fix for KAFKA-15608 in 2023; it was closed by its author
during the reproduction discussion, reopened by a reviewer the same day, and
eventually closed by the stale bot in Jan 2025 with the reproduction still
unconfirmed. Its test was never annotated with @Test, and the assertion helper
returned true when the cache was empty, so it would have passed against the
unfixed code either way.
I have a deterministic unit-level reproduction on trunk and a fix that carries
the epoch history over on the swap: https://github.com/apache/kafka/pull/23205
> Loop of OffsetOutOfRangeException
> ---------------------------------
>
> Key: KAFKA-20961
> URL: https://issues.apache.org/jira/browse/KAFKA-20961
> Project: Kafka
> Issue Type: Bug
> Components: replication
> Affects Versions: 4.1.1
> Reporter: David Dufour
> Priority: Major
>
> Hello,
> we have identified scenarios where a leader broker returns
> OffsetOutOfRangeException error to the leader in loop.
> This is probably what was described in KAFKA-15608 but this issue was logged
> on an old version and is still in status open.
> We are experiencing this issue very regularly as we are using JBOD and we
> move partitions from one disk to another inside a same broker.
> Here is an example of logs:
> {+}Leader:
> {+}INFO Attempting to replace current log Log(...) with Log(...) for xxxxxx
> (kafka.log.LogManager) [ReplicaAlterLogDirsThread-3]
> ...\{+}
> {+}INFO The current replica is successfully replaced with the future replica
> for xxxxxx (kafka.log.LogManager) [ReplicaAlterLogDirsThread-3]
> +Follower, few ms later:+
> INFO [ReplicaFetcher replicaId=1, leaderId=3, fetcherId=0] Reset fetch offset
> for partition xxxxxx from 0 to the current local replica's end offset 0
> (kafka.server.ReplicaFetcherThread) [ReplicaFetcherThread-0-3]
> INFO [ReplicaFetcher replicaId=1, leaderId=3, fetcherId=0] Current offset 0
> for partition xxxxxx is out of range, which typically implies a leader
> change. Reset fetch offset to 0 (kafka.server.ReplicaFetcherThread)
> [ReplicaFetcherThread-0-3]
> ....repeat every ~12ms
> Often, the loop stops by itself after some time (traffic injected, broker
> restart,...).
> We use Cruise Control to perform cluster rebalance. Recently, a rebalance
> triggered a huge number of such loops that never stopped. The logs were
> flooded and in the end, the cluster became unresponsive to the clients,
> causing a severe outage. A rolling restart recovered the situation.
> +How to reproduce:+
> 1- create new empty topic
> 2- build some leader epoch (for ex: switch leadership from one broker to
> another)
> 3- move partition on leader from dir1 to dir2
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
