Avishek Das created KAFKA-21098:
-----------------------------------
Summary: RemoteDeleteLagSegments/RemoteDeleteLagBytes report a
phantom lag that never drains after a leader→follower transition
Key: KAFKA-21098
URL: https://issues.apache.org/jira/browse/KAFKA-21098
Project: Kafka
Issue Type: Bug
Components: Tiered-Storage
Affects Versions: 4.3.1, 4.2.1, 4.1.2, 4.0.2, 3.9.2
Reporter: Avishek Das
Assignee: Avishek Das
h2. Summary
On a tiered-storage broker, the per-topic {{RemoteDeleteLagSegments}} (and its
byte twin {{{}RemoteDeleteLagBytes{}}}) gauge can get latched at a non-zero
value that never drains back to 0, even though remote deletion is healthy and
there is no orphaned segment in remote storage. The phantom value appears when
a partition leader moves to another broker (any leadership change — planned
reassignment, preferred-leader election, or a rebalance) while an expiration
pass for that partition is still in flight on the old leader. It is purely a
metrics artifact: there is no real backlog and no data loss. Frequent
leadership movement makes the broker/cluster aggregate step up over time as
each affected partition contributes a stuck residual.
h2. Affects
Any tiered-storage-enabled topic in 3.9.0 through trunk (reproduced on 4.2.1).
The likelihood scales with leadership-change frequency: clusters that reassign
or rebalance leadership often (e.g. automated balancers, rolling restarts, or
under-provisioned brokers where leadership shifts a lot) accumulate more stuck
residuals.
h2. Root cause
The copy path was hardened for this exact race (KAFKA-16948: reset lag metrics
on becoming follower, and guard {{recordLagStats}} with
{{{}!isCancelled(){}}}), but the delete path was never given the equivalent
treatment. Two gaps combine:
(1) *No cancellation guard on the emit.*
{{RLMExpirationTask.updateRemoteDeleteLagWith(...)}} writes
{{{}RemoteDeleteLagSegments{}}}/{{{}RemoteDeleteLagBytes{}}} unconditionally.
There is no {{if (!isCancelled())}} check — unlike the copy path's
{{{}recordLagStats{}}}, which already skips emitting for a cancelled task.
(2) *The removal and the in-flight emit race.* On a leader→follower transition
{{onLeadershipChange}} runs {{doHandleFollowerPartition}} (which cancels the
{{{}RLMExpirationTask{}}}) and then {{removeRemoteTopicPartitionMetrics}}
(which removes the delete-lag gauges). But an expiration-pool thread that is
already mid-{{{}cleanupExpiredRemoteLogSegments{}}} for that partition can call
{{updateRemoteDeleteLagWith(n, bytes)}} *after* the gauge has been removed,
re-registering it at a non-zero value.
Because the same pass then reaches {{{}deleteRemoteLogSegment(..., ignored ->
!isCancelled()){}}}, which returns {{false}} for the now-cancelled task, the
segment is *not* deleted by this old leader and the counter is never
decremented back down. The gauge is therefore pinned at the re-registered value
until the same broker re-leads that partition and completes a fresh expiration
pass, the partition is stopped, or the broker restarts.
Note the underlying segment is *not* orphaned: the new leader independently
re-evaluates retention and deletes it. So the only symptom is a stuck metric /
false-positive alert.
h2. Worked example
Config: a tiered-storage topic ({{remote.storage.enable=true}}) with short
retention so expiration runs continuously, and a setup where leadership for one
of its partitions moves off broker A to broker B mid-pass.
*Phase A — steady state on broker A (leader):* each expiration cycle sets
{{RemoteDeleteLagSegments}} to the batch of retention-breached segments at pass
start, then decrements to 0 as each segment is deleted. Healthy sawtooth, floor
= 0.
*Phase B — leadership moves off broker A mid-pass:*
|| Step || Thread || Action || Delete-lag gauge on A ||
| 1 | expiration pool | pass starts, finds 1 breached segment,
{{updateRemoteDeleteLagWith(1, sz)}} | 1 |
| 2 | leadership handler | {{onLeadershipChange}} → cancel
{{RLMExpirationTask}} for the partition | 1 |
| 3 | leadership handler | {{removeRemoteTopicPartitionMetrics}} removes the
gauge | (removed) |
| 4 | expiration pool | still in the same pass, {{updateRemoteDeleteLagWith(1,
sz)}} runs again | 1 (re-registered) |
| 5 | expiration pool | {{deleteRemoteLogSegment(..., ignored ->
!isCancelled())}} returns false → no delete, no decrement | 1 (stuck) |
The gauge on broker A now stays at 1 forever. Under frequent leadership churn
the broker/cluster aggregate steps up monotonically as each quiet partition
contributes its stuck 1.
h2. Steps to reproduce
# Enable tiered storage on a topic with short retention so the expiration task
deletes segments regularly.
# Drive enough produce traffic that segments roll and become retention-breached.
# Repeatedly move leadership of the partition to another broker while
expiration is active (reassignment / preferred-leader election / rebalance).
# Observe
{{kafka.server:type=BrokerTopicMetrics,name=RemoteDeleteLagSegments,topic=<topic>}}
on the former leader: it stays pinned at a non-zero value with no
{{RemoteDeleteErrors}} and no {{RemoteStorageException}}, and does not drain to
0.
h2. Expected vs actual
* *Expected:* {{RemoteDeleteLagSegments}}/{{RemoteDeleteLagBytes}} return to 0
on a broker once it is no longer the leader for the partition (the new leader
owns deletion), mirroring the copy-lag behavior fixed in KAFKA-16948.
* *Actual:* the metric stays latched at a non-zero value (typically 1) on the
former leader indefinitely, producing false-positive "delete lag stuck" alerts.
h2. Proposed fix
Mirror the copy path. Guard the emit with the task's cancellation state so a
cancelled expiration task cannot re-register the gauge after
{{removeRemoteTopicPartitionMetrics}} has cleared it:
{code:java}
// VisibleForTesting
void updateRemoteDeleteLagWith(int segmentsLeftToDelete, long
sizeOfDeletableSegmentsBytes) {
if (!isCancelled()) {
String topic = topicIdPartition.topic();
int partition = topicIdPartition.partition();
brokerTopicStats.recordRemoteDeleteLagSegments(topic, partition,
segmentsLeftToDelete);
brokerTopicStats.recordRemoteDeleteLagBytes(topic, partition,
sizeOfDeletableSegmentsBytes);
}
}
{code}
This closes the race in the same way KAFKA-16948 did for {{recordLagStats}} on
the copy path. Fix proposed in PR #<TBD>.
h2. Related issues
* KAFKA-16948 — reset lag metrics on becoming follower (guarded the *copy*
path's {{recordLagStats}}); this bug is the un-fixed *delete* path equivalent.
* KAFKA-15147 — measure pending and outstanding remote segment operations
(umbrella that introduced the RemoteCopyLag/RemoteDeleteLag metrics).
* KAFKA-20977 — a separate phantom-lag off-by-one on the copy path
(single-record segments); different root cause, same "gauge stuck non-zero"
symptom class.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
