gortiz opened a new issue, #19970:
URL: https://github.com/apache/pinot/issues/19970
## Summary
`pinot_controller_offlineSegmentDelayHours` (and `lastPushTimeDelayHours`)
can report a delay up to one checker period (24h by default) higher than the
real delay. The gauge computes `now - stored max segment end time` on each
scrape, which is correct. But the stored value only changes when
`OfflineSegmentValidationManager` runs. A segment uploaded right after a run
has no effect on the gauge until the next run.
## How it works today
- `OfflineSegmentValidationManager.validateOfflineSegmentPush()` reads the
ZK metadata of all segments of the table, computes the max `endTimeMs` and max
push time, and stores them with
`ValidationMetrics.updateOfflineSegmentDelayGauge()` /
`updateLastPushTimeGauge()`.
- `ValidationMetrics.CurrentTimeMillisDeltaGaugeHours.value()` returns
`(System.currentTimeMillis() - storedValue) / 1h` on each read.
- The task runs every
`controller.offline.segment.interval.checker.frequencyPeriod`, which is `24h`
by default.
As a result, the gauge is a sawtooth: between two runs it increases by 1h
every hour, even when segments arrive on time.
```
reported delay
| /| /| /|
| / | / | / | <- peak = real delay + up to 24h
| / | / | / |
|/ |/ |/ | <- value right after each run = real
delay
+-------+-------+-------+----> time
run run run
```
## Example
A hybrid table gets one OFFLINE segment per day from SegmentImportTask at
00:20 UTC. The validation task runs at 14:32 UTC.
- At 14:07 UTC the newest segment was already in ZK, uploaded at 00:20 UTC,
with an end time about 70h old.
- The gauge still used the end time of the previous segment and reported
96.05h.
- At the 14:32 UTC run the gauge dropped to 69.98h.
An alert at `>= 96` fired for 25 minutes on data that was not late. For 30
days the daily peak was 92–95h and the real delay was 68–72h on every day.
## Suggested fix
This is a suggestion. Other approaches are fine.
Update the stored max end time and max push time when the controller accepts
a new or refreshed OFFLINE segment. The upload path already has the new
`SegmentZKMetadata`, so the update is `stored = max(stored, newEndTimeMs)` with
no ZK scan. `OfflineSegmentValidationManager` keeps its full recompute, which
corrects the value after segment deletion or retention, when the max can
decrease.
Alternatives:
- Lower the default checker period. This is configuration only, but it adds
ZK reads for clusters with many segments, and it reduces the error without
removing it.
- Document the behavior and tell users to alert on
`min_over_time(...[<checker period>])`. This is what we do today as a
mitigation.
## Workaround
Alert on
`min_over_time(pinot_controller_offlineSegmentDelayHours_Value[24h])`, which
reads the value right after the last run. The cost is that a real stall shows
up to 24h later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
