[
https://issues.apache.org/jira/browse/KAFKA-21098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kamal Chandraprakash resolved KAFKA-21098.
------------------------------------------
Fix Version/s: 4.5.0
Resolution: Fixed
> RemoteDeleteLagSegments/RemoteDeleteLagBytes report a phantom lag that never
> drains after a leader→follower transition
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-21098
> URL: https://issues.apache.org/jira/browse/KAFKA-21098
> Project: Kafka
> Issue Type: Bug
> Components: Tiered-Storage
> Affects Versions: 3.9.2, 4.0.2, 4.1.2, 4.2.1, 4.3.1
> Reporter: Avishek Das
> Assignee: Avishek Das
> Priority: Minor
> Labels: tiered-storage
> Fix For: 4.5.0
>
>
> h2. Summary
> On a tiered-storage broker, the per-topic {{RemoteDeleteLagSegments}} (and
> its byte twin {{{}RemoteDeleteLagBytes{}}}) gauge can get latched at a
> non-zero value that never drains back to 0, even though remote deletion is
> healthy and there is no orphaned segment in remote storage. The phantom value
> appears when a partition leader moves to another broker (any leadership
> change — planned reassignment, preferred-leader election, or a rebalance)
> while an expiration pass for that partition is still in flight on the old
> leader. It is purely a metrics artifact: there is no real backlog and no data
> loss. Frequent leadership movement makes the broker/cluster aggregate step up
> over time as each affected partition contributes a stuck residual.
> h2. Affects
> Any tiered-storage-enabled topic in 3.9.0 through trunk (reproduced on
> 4.2.1). The likelihood scales with leadership-change frequency: clusters that
> reassign or rebalance leadership often (e.g. automated balancers, rolling
> restarts, or under-provisioned brokers where leadership shifts a lot)
> accumulate more stuck residuals.
> h2. Root cause
> The copy path was hardened for this exact race (KAFKA-16948: reset lag
> metrics on becoming follower, and guard {{recordLagStats}} with
> {{{}!isCancelled(){}}}), but the delete path was never given the equivalent
> treatment. Two gaps combine:
> (1) *No cancellation guard on the emit.*
> {{RLMExpirationTask.updateRemoteDeleteLagWith(...)}} writes
> {{{}RemoteDeleteLagSegments{}}}/{{{}RemoteDeleteLagBytes{}}} unconditionally.
> There is no {{if (!isCancelled())}} check — unlike the copy path's
> {{{}recordLagStats{}}}, which already skips emitting for a cancelled task.
> (2) *The removal and the in-flight emit race.* On a leader→follower
> transition {{onLeadershipChange}} runs {{doHandleFollowerPartition}} (which
> cancels the {{{}RLMExpirationTask{}}}) and then
> {{removeRemoteTopicPartitionMetrics}} (which removes the delete-lag gauges).
> But an expiration-pool thread that is already
> mid-{{{}cleanupExpiredRemoteLogSegments{}}} for that partition can call
> {{updateRemoteDeleteLagWith(n, bytes)}} *after* the gauge has been removed,
> re-registering it at a non-zero value.
> Because the same pass then reaches {{{}deleteRemoteLogSegment(..., ignored ->
> !isCancelled()){}}}, which returns {{false}} for the now-cancelled task, the
> segment is *not* deleted by this old leader and the counter is never
> decremented back down. The gauge is therefore pinned at the re-registered
> value until the same broker re-leads that partition and completes a fresh
> expiration pass, the partition is stopped, or the broker restarts.
> Note the underlying segment is *not* orphaned: the new leader independently
> re-evaluates retention and deletes it. So the only symptom is a stuck metric
> / false-positive alert.
> h2. Worked example
> Config: a tiered-storage topic ({{remote.storage.enable=true}}) with short
> retention so expiration runs continuously, and a setup where leadership for
> one of its partitions moves off broker A to broker B mid-pass.
> *Phase A — steady state on broker A (leader):* each expiration cycle sets
> {{RemoteDeleteLagSegments}} to the batch of retention-breached segments at
> pass start, then decrements to 0 as each segment is deleted. Healthy
> sawtooth, floor = 0.
> *Phase B — leadership moves off broker A mid-pass:*
> || Step || Thread || Action || Delete-lag gauge on A ||
> | 1 | expiration pool | pass starts, finds 1 breached segment,
> {{updateRemoteDeleteLagWith(1, sz)}} | 1 |
> | 2 | leadership handler | {{onLeadershipChange}} → cancel
> {{RLMExpirationTask}} for the partition | 1 |
> | 3 | leadership handler | {{removeRemoteTopicPartitionMetrics}} removes the
> gauge | (removed) |
> | 4 | expiration pool | still in the same pass,
> {{updateRemoteDeleteLagWith(1, sz)}} runs again | 1 (re-registered) |
> | 5 | expiration pool | {{deleteRemoteLogSegment(..., ignored ->
> !isCancelled())}} returns false → no delete, no decrement | 1 (stuck) |
> The gauge on broker A now stays at 1 forever. Under frequent leadership churn
> the broker/cluster aggregate steps up monotonically as each quiet partition
> contributes its stuck 1.
> h2. Steps to reproduce
> # Enable tiered storage on a topic with short retention so the expiration
> task deletes segments regularly.
> # Drive enough produce traffic that segments roll and become
> retention-breached.
> # Repeatedly move leadership of the partition to another broker while
> expiration is active (reassignment / preferred-leader election / rebalance).
> # Observe
> {{kafka.server:type=BrokerTopicMetrics,name=RemoteDeleteLagSegments,topic=<topic>}}
> on the former leader: it stays pinned at a non-zero value with no
> {{RemoteDeleteErrors}} and no {{RemoteStorageException}}, and does not drain
> to 0.
> h2. Expected vs actual
> * *Expected:* {{RemoteDeleteLagSegments}}/{{RemoteDeleteLagBytes}} return to
> 0 on a broker once it is no longer the leader for the partition (the new
> leader owns deletion), mirroring the copy-lag behavior fixed in KAFKA-16948.
> * *Actual:* the metric stays latched at a non-zero value (typically 1) on the
> former leader indefinitely, producing false-positive "delete lag stuck"
> alerts.
> h2. Proposed fix
> Mirror the copy path. Guard the emit with the task's cancellation state so a
> cancelled expiration task cannot re-register the gauge after
> {{removeRemoteTopicPartitionMetrics}} has cleared it:
> {code:java}
> // VisibleForTesting
> void updateRemoteDeleteLagWith(int segmentsLeftToDelete, long
> sizeOfDeletableSegmentsBytes) {
> if (!isCancelled()) {
> String topic = topicIdPartition.topic();
> int partition = topicIdPartition.partition();
> brokerTopicStats.recordRemoteDeleteLagSegments(topic, partition,
> segmentsLeftToDelete);
> brokerTopicStats.recordRemoteDeleteLagBytes(topic, partition,
> sizeOfDeletableSegmentsBytes);
> }
> }
> {code}
> This closes the race in the same way KAFKA-16948 did for {{recordLagStats}}
> on the copy path. Fix proposed in PR
> https://github.com/apache/kafka/pull/23471.
> h2. Related issues
> * KAFKA-16948 — reset lag metrics on becoming follower (guarded the *copy*
> path's {{recordLagStats}}); this bug is the un-fixed *delete* path equivalent.
> * KAFKA-15147 — measure pending and outstanding remote segment operations
> (umbrella that introduced the RemoteCopyLag/RemoteDeleteLag metrics).
> * KAFKA-20977 — a separate phantom-lag off-by-one on the copy path
> (single-record segments); different root cause, same "gauge stuck non-zero"
> symptom class.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
