swaminathanmanish opened a new pull request, #19632: URL: https://github.com/apache/pinot/pull/19632
## Problem The per-partition `llcPartitionConsuming` server gauge is set to `1` while a segment is consuming and to `0` when it stops. For a normal segment commit a new consuming segment is created and flips the gauge back to `1`. But when a stream partition is **retired** — e.g. a Kinesis shard is split/merged and fully consumed (end of partition group) — no successor consuming segment is ever created, so the gauge is left at `0` and **lingers there forever** (the gauge key is `table-topic-partition`, so nothing overwrites it). Monitoring built on this gauge (for example an alert that fires when `llcPartitionConsuming == 0`) then fires indefinitely for a partition that is legitimately done consuming — pure noise. ## Root cause `RealtimeSegmentDataManager` sets the gauge to `0` in several stop paths, including `goOnlineFromConsuming()` (the CONSUMING→ONLINE transition). It is only *removed* by `cleanupMetrics()`, which runs on host-swap / partition-count change — not on shard retirement, where the committed ONLINE segment stays hosted on the server. ## Fix Add a small `markConsumingStopped()` helper: when consumption stops because the partition reached end of partition group (`_endOfPartitionGroup`), **remove** the gauge so the series goes absent; otherwise set it to `0` as before. It replaces the `setValueOfTableGauge(..., LLC_PARTITION_CONSUMING, 0)` calls on the end-of-partition-group stop paths (`goOnlineFromConsuming`, the consume-loop exit, and the catch-up / replace `finally` blocks). The ERROR path still sets `0` so genuine failures stay visible. ## Behaviour preserved - Normal commits still set `0` and get flipped back to `1` by the successor consuming segment. - Errors and holds still report `0`. - Only genuinely-retired (end-of-partition-group) partitions have their gauge removed. ## Testing `RealtimeSegmentDataManagerTest`: - `testEndOfPartitionGroupRemovesConsumingGauge` — with `_endOfPartitionGroup` set, `goOnlineFromConsuming` **removes** the gauge and never sets it to `0`. - `testNormalCommitKeepsConsumingGaugeAtZero` — a normal commit **sets** the gauge to `0` and never removes it. Both fail without this change and pass with it; the existing `testOnlineTransitionAfterStop` still passes. ## Note / possible follow-up The gauge is removed on every replica that consumed to the shard end (the common case for a closed shard). A replica that *downloaded* the last segment instead of consuming to its end would not have `_endOfPartitionGroup` set locally; a follow-up can propagate the end-of-partition-group marker via the committed segment's ZK metadata custom map so those replicas also remove the gauge. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
