José Armando García Sancio created KAFKA-14932:
--------------------------------------------------
Summary: Heuristic for increasing the log start offset after
replicas are caught up
Key: KAFKA-14932
URL: https://issues.apache.org/jira/browse/KAFKA-14932
Project: Kafka
Issue Type: Sub-task
Reporter: José Armando García Sancio
Assignee: José Armando García Sancio
The implementation in [https://github.com/apache/kafka/pull/9816] increases the
log start offset as soon as a snapshot is created that is greater than the log
start offset. This is correct but causes some inefficiency in some cases.
# Any follower, voters or observers, with an end offset between the leader's
log start offset and the leader's latest snapshot will get invalidated. This
will cause those follower to fetch the new snapshot and reload it's state
machine.
# Any {{Listener}} or state machine that has a {{nextExpectedOffset()}} less
than the latest snapshot will get invalidated. This will cause the state
machine to have to reload its state from the latest snapshot.
To minimize the frequency of these reloads KIP-630 proposes adding the
following configuration:
* {{metadata.start.offset.lag.time.max.ms}} - The maximum amount of time that
leader will wait for an offset to get replicated to all of the live replicas
before advancing the {{{}LogStartOffset{}}}. See section “When to Increase the
LogStartOffset”. The default is 7 days.
This description and implementation should be extended to also apply to the
state machine, or {{{}Listener{}}}. The local log start offset should be
increased when all of the {{{}ListenerContext{}}}'s {{nextExpectedOffset()}} is
greater than the offset of the latest snapshot.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
