swaminathanmanish opened a new pull request, #19094:
URL: https://github.com/apache/pinot/pull/19094
## Problem
`SegmentStatusChecker` computes `percentOfReplicas` (which drives the
`SegmentReplicasCriticallyLowForHATable` alert) as a min across segments of
`replicasUpInEV / replicasInIS`. It skips just-created/pushed segments for a
grace window (`controller.statuschecker.waitForPushTimePeriod`) so servers have
time to load them. The grace timestamp was derived as:
```java
long creationTimeMs = segmentZKMetadata.getStatus() == Status.IN_PROGRESS
? segmentZKMetadata.getCreationTime()
: segmentZKMetadata.getPushTime();
```
Real-time (LLC) committed segments **never populate push time** (it stays
`Long.MIN_VALUE`), so a just-committed real-time segment is never graced — it
is included in the replica check the moment it commits, while its replicas are
still loading.
**Pauseless ingestion makes this a frequent false positive.** A pauseless
segment enters the `COMMITTING` status ("done consuming, immutable segment not
yet committed") and its replicas rebuild/reload the immutable segment for the
build window (tens of seconds to a few minutes). During that window it is
transiently under-replicated in the ExternalView, but `COMMITTING` was treated
as non-`IN_PROGRESS` and checked immediately. Because the gauge is a min across
segments, one just-committed segment drags the table's `percentOfReplicas` down
and fires the alert even though the table is fully redundant. High-ingest
tables (a segment sealing every few minutes per partition) page repeatedly.
## Fix
Key the grace window on a timestamp that is actually populated for these
segments:
- `IN_PROGRESS` and `COMMITTING` → creation time
- committed/pushed → push time when set, else fall back to creation time
(covers LLC committed `DONE` segments)
The existing `waitForPushTime` window is reused (no new config). Once a
segment is older than the window and still under-replicated, it is checked
normally — so genuinely stuck commits and real replica losses still alert.
## Tests
Added to `SegmentStatusCheckerTest`:
- `realtimeCommittingSegmentWithinGraceNotUnderReplicated` — a `COMMITTING`
segment at 1/3 replicas ONLINE, just created, within the grace window →
`percentOfReplicas` stays 100, `segmentsWithLessReplicas` = 0 (the false-alert
case is now graced).
- `realtimeCommittingSegmentBeyondGraceUnderReplicated` — same segment
created 2h ago (past the grace) → `percentOfReplicas` drops to 33 and
`segmentsWithLessReplicas` = 1 (a stuck commit is still flagged).
Existing `SegmentStatusCheckerTest` cases pass unchanged (the change is
behavior-preserving for them).
## Release Notes
Fixes false `SegmentReplicasCriticallyLowForHATable` / low
`percentOfReplicas` alerts caused by real-time committed and pauseless
`COMMITTING` segments being counted as under-replicated during their normal
post-commit load window.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
