Hello Team, Could someone from the PMC please take a look at this Kafka tiered storage bug PR and review it when you get a chance?
Thanks! On Tue, Sep 15, 2026 at 6:24 PM Avishek Das (Jira) <[email protected]> wrote: > Avishek Das created KAFKA-21098: > ----------------------------------- > > Summary: RemoteDeleteLagSegments/RemoteDeleteLagBytes report > a phantom lag that never drains after a leader→follower transition > Key: KAFKA-21098 > URL: https://issues.apache.org/jira/browse/KAFKA-21098 > Project: Kafka > Issue Type: Bug > Components: Tiered-Storage > Affects Versions: 4.3.1, 4.2.1, 4.1.2, 4.0.2, 3.9.2 > Reporter: Avishek Das > Assignee: Avishek Das > > > h2. Summary > > On a tiered-storage broker, the per-topic {{RemoteDeleteLagSegments}} (and > its byte twin {{{}RemoteDeleteLagBytes{}}}) gauge can get latched at a > non-zero value that never drains back to 0, even though remote deletion is > healthy and there is no orphaned segment in remote storage. The phantom > value appears when a partition leader moves to another broker (any > leadership change — planned reassignment, preferred-leader election, or a > rebalance) while an expiration pass for that partition is still in flight > on the old leader. It is purely a metrics artifact: there is no real > backlog and no data loss. Frequent leadership movement makes the > broker/cluster aggregate step up over time as each affected partition > contributes a stuck residual. > h2. Affects > > Any tiered-storage-enabled topic in 3.9.0 through trunk (reproduced on > 4.2.1). The likelihood scales with leadership-change frequency: clusters > that reassign or rebalance leadership often (e.g. automated balancers, > rolling restarts, or under-provisioned brokers where leadership shifts a > lot) accumulate more stuck residuals. > h2. Root cause > > The copy path was hardened for this exact race (KAFKA-16948: reset lag > metrics on becoming follower, and guard {{recordLagStats}} with > {{{}!isCancelled(){}}}), but the delete path was never given the equivalent > treatment. Two gaps combine: > > (1) *No cancellation guard on the emit.* > {{RLMExpirationTask.updateRemoteDeleteLagWith(...)}} writes > {{{}RemoteDeleteLagSegments{}}}/{{{}RemoteDeleteLagBytes{}}} > unconditionally. There is no {{if (!isCancelled())}} check — unlike the > copy path's {{{}recordLagStats{}}}, which already skips emitting for a > cancelled task. > > (2) *The removal and the in-flight emit race.* On a leader→follower > transition {{onLeadershipChange}} runs {{doHandleFollowerPartition}} (which > cancels the {{{}RLMExpirationTask{}}}) and then > {{removeRemoteTopicPartitionMetrics}} (which removes the delete-lag > gauges). But an expiration-pool thread that is already > mid-{{{}cleanupExpiredRemoteLogSegments{}}} for that partition can call > {{updateRemoteDeleteLagWith(n, bytes)}} *after* the gauge has been removed, > re-registering it at a non-zero value. > > Because the same pass then reaches {{{}deleteRemoteLogSegment(..., ignored > -> !isCancelled()){}}}, which returns {{false}} for the now-cancelled task, > the segment is *not* deleted by this old leader and the counter is never > decremented back down. The gauge is therefore pinned at the re-registered > value until the same broker re-leads that partition and completes a fresh > expiration pass, the partition is stopped, or the broker restarts. > > Note the underlying segment is *not* orphaned: the new leader > independently re-evaluates retention and deletes it. So the only symptom is > a stuck metric / false-positive alert. > > h2. Worked example > > Config: a tiered-storage topic ({{remote.storage.enable=true}}) with short > retention so expiration runs continuously, and a setup where leadership for > one of its partitions moves off broker A to broker B mid-pass. > > *Phase A — steady state on broker A (leader):* each expiration cycle sets > {{RemoteDeleteLagSegments}} to the batch of retention-breached segments at > pass start, then decrements to 0 as each segment is deleted. Healthy > sawtooth, floor = 0. > > *Phase B — leadership moves off broker A mid-pass:* > || Step || Thread || Action || Delete-lag gauge on A || > | 1 | expiration pool | pass starts, finds 1 breached segment, > {{updateRemoteDeleteLagWith(1, sz)}} | 1 | > | 2 | leadership handler | {{onLeadershipChange}} → cancel > {{RLMExpirationTask}} for the partition | 1 | > | 3 | leadership handler | {{removeRemoteTopicPartitionMetrics}} removes > the gauge | (removed) | > | 4 | expiration pool | still in the same pass, > {{updateRemoteDeleteLagWith(1, sz)}} runs again | 1 (re-registered) | > | 5 | expiration pool | {{deleteRemoteLogSegment(..., ignored -> > !isCancelled())}} returns false → no delete, no decrement | 1 (stuck) | > > The gauge on broker A now stays at 1 forever. Under frequent leadership > churn the broker/cluster aggregate steps up monotonically as each quiet > partition contributes its stuck 1. > > h2. Steps to reproduce > > # Enable tiered storage on a topic with short retention so the expiration > task deletes segments regularly. > # Drive enough produce traffic that segments roll and become > retention-breached. > # Repeatedly move leadership of the partition to another broker while > expiration is active (reassignment / preferred-leader election / rebalance). > # Observe > {{kafka.server:type=BrokerTopicMetrics,name=RemoteDeleteLagSegments,topic=<topic>}} > on the former leader: it stays pinned at a non-zero value with no > {{RemoteDeleteErrors}} and no {{RemoteStorageException}}, and does not > drain to 0. > > h2. Expected vs actual > > * *Expected:* {{RemoteDeleteLagSegments}}/{{RemoteDeleteLagBytes}} return > to 0 on a broker once it is no longer the leader for the partition (the new > leader owns deletion), mirroring the copy-lag behavior fixed in KAFKA-16948. > * *Actual:* the metric stays latched at a non-zero value (typically 1) on > the former leader indefinitely, producing false-positive "delete lag stuck" > alerts. > > > h2. Proposed fix > > Mirror the copy path. Guard the emit with the task's cancellation state so > a cancelled expiration task cannot re-register the gauge after > {{removeRemoteTopicPartitionMetrics}} has cleared it: > > {code:java} > // VisibleForTesting > void updateRemoteDeleteLagWith(int segmentsLeftToDelete, long > sizeOfDeletableSegmentsBytes) { > if (!isCancelled()) { > String topic = topicIdPartition.topic(); > int partition = topicIdPartition.partition(); > brokerTopicStats.recordRemoteDeleteLagSegments(topic, partition, > segmentsLeftToDelete); > brokerTopicStats.recordRemoteDeleteLagBytes(topic, partition, > sizeOfDeletableSegmentsBytes); > } > } > {code} > > This closes the race in the same way KAFKA-16948 did for > {{recordLagStats}} on the copy path. Fix proposed in PR #<TBD>. > > h2. Related issues > > * KAFKA-16948 — reset lag metrics on becoming follower (guarded the *copy* > path's {{recordLagStats}}); this bug is the un-fixed *delete* path > equivalent. > * KAFKA-15147 — measure pending and outstanding remote segment operations > (umbrella that introduced the RemoteCopyLag/RemoteDeleteLag metrics). > * KAFKA-20977 — a separate phantom-lag off-by-one on the copy path > (single-record segments); different root cause, same "gauge stuck non-zero" > symptom class. > > > > > > -- > This message was sent by Atlassian Jira > (v8.20.10#820010) > -- Thanks, *Avishek Das* <https://www.linkedin.com/in/imavishek/> Member Of Technical Staff At Salesforce <https://www.linkedin.com/company/salesforce/> *[email protected] <[email protected]>* Mobile: +91-7008383890, +91-8908904383
