Jax created KAFKA-21190:
---------------------------
Summary: Empty follower fails to start when the KIP-1023 fetch
start offset equals the leader's log start offset
Key: KAFKA-21190
URL: https://issues.apache.org/jira/browse/KAFKA-21190
Project: Kafka
Issue Type: Bug
Components: Tiered-Storage
Affects Versions: 4.3.0
Reporter: Jax
Assignee: Jax
h3. Problem
With {{{}follower.fetch.last.tiered.offset.enable=true{}}}, an empty follower
can select a fetch start offset that equals the leader's log start offset. In
that case the remote range to rebuild auxiliary state from, [logStartOffset,
fetchStartOffset), is empty.
{{TierStateMachine.buildRemoteLogAuxState}} still looks up remote segment
metadata for {{{}fetchStartOffset - 1{}}}. That offset is below the log start
offset, and its remote log segment may already be deleted, so the lookup can
return empty. When it does, the follower fails with
{{{}RemoteStorageException{}}}, retries, and stays out of the ISR until the
selected offset changes (for example, after the next segment upload). On a
low-volume topic this can take a long time.
h3. Observed on 4.3.0
We rebuilt a follower on an empty disk. The selected fetch start offset and the
leader's log start offset were both 18203:
{noformat}
ERROR [ReplicaFetcher replicaId=0, leaderId=7, fetcherId=0] Error getting
offset for partition <TOPIC-PARTITION>
org.apache.kafka.server.log.remote.storage.RemoteStorageException: Couldn't
build the state from remote store for partition: <TOPIC-PARTITION>,
currentLeaderEpoch: 406, leaderLocalLogStartOffset: 18203,
leaderLogStartOffset: 18203, epoch: 405 as the previous remote log segment
metadata was not found
at
kafka.server.TierStateMachine.buildRemoteStorageException(TierStateMachine.java:278)
at
kafka.server.TierStateMachine.lambda$buildRemoteLogAuxState$0(TierStateMachine.java:237)
at java.base/java.util.Optional.orElseThrow(Optional.java:403)
at
kafka.server.TierStateMachine.buildRemoteLogAuxState(TierStateMachine.java:237)
at kafka.server.TierStateMachine.start(TierStateMachine.java:113)
at
kafka.server.AbstractFetcherThread.fetchOffsetAndTruncate(AbstractFetcherThread.scala:679)
at
kafka.server.AbstractFetcherThread.handleOutOfRangeError(AbstractFetcherThread.scala:749)
{noformat}
On this call path, the {{leaderLocalLogStartOffset}} field in the message is
the selected fetch start offset.
Dynamically setting {{follower.fetch.last.tiered.offset.enable=false}} bypassed
this path and the follower recovered without a restart.
h3. Code
[TierStateMachine.java#L201-L238|https://github.com/apache/kafka/blob/4.3.0/core/src/main/java/kafka/server/TierStateMachine.java#L201-L238]
(same on trunk):
{code:java}
long previousOffsetToLeaderLocalLogStartOffset = leaderLocalLogStartOffset - 1;
...
RemoteLogSegmentMetadata remoteLogSegmentMetadata =
rlm.fetchRemoteLogSegmentMetadata(topicPartition, targetEpoch,
previousOffsetToLeaderLocalLogStartOffset)
.orElseThrow(() -> buildRemoteStorageException(...));
{code}
{{UnifiedLog.fetchEarliestPendingUploadOffset}} clamps its result to
{{{}max(highestRemoteOffset + 1, logStartOffset){}}}, and
{{AbstractFetcherThread.fetchEarliestPendingUploadOffset}} returns the local
log start offset when nothing is uploaded and it equals the log start offset.
So {{TierStateMachine}} can receive equal offsets during normal operation.
KIP-1023 describes this case as a normal local fetch: "it simply means that
there are no valid log segments on the remote yet. Hence the ReplicaFetcher
simply truncates its log and starts replicating all data the leader has
locally."
h3. Test gap
{{testLastTieredOffsetWithNonZeroLSOOnNewLeader}} and
{{testLastTieredOffsetWithSlowUploadNoLocalDeletion}} in
{{AbstractFetcherThreadTest}} cover equal positive start offsets, but both use
{{{}MockTierStateMachine{}}}, which truncates directly to the selected offset
and skips the metadata lookup. No test runs the real {{TierStateMachine}} in
this case.
h3. Proposed fix
In {{{}TierStateMachine.buildRemoteLogAuxState{}}}, when the fetch start offset
equals the leader's log start offset, skip the earlier-epoch and remote
metadata lookups, truncate fully to the leader's log start offset, and fetch
from there. Keep the existing error when the remote range is not empty.
h3. Related
* KIP-1023 discussion:
[https://lists.apache.org/thread/ypwqnwyn7zn6xo781yjq4v99p44gv62h]
* Question on dev@:
[https://lists.apache.org/thread/94hv8gsn7hqsrsn2os7nh6qxz6lwp6y1]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
