noob-se7en opened a new pull request, #18942:
URL: https://github.com/apache/pinot/pull/18942
## Description
Server-level realtime consumption rate limiting had two observability gaps:
1. The server-wide quota utilization was published through the per-table
`CONSUMPTION_QUOTA_UTILIZATION` gauge keyed by the meter name
`realtimeRowsConsumed`. Being a server-wide value, the JMX/Prometheus exporters
map it onto a misleading, per-table-shaped series that server-level dashboards
cannot match.
2. `QuotaUtilizationTracker`/`AsyncMetricEmitter` aggregated the per-minute
unit count in an `int` and cast the `LongAdder` sum to `int` in `emit()`. In
byte-based throttling mode the per-minute byte sum for a busy server exceeds
`Integer.MAX_VALUE` (~35.8 MB/s sustained over the 60s window), overflowing to
a negative utilization.
There was also no metric exposing the configured rate limit, so a dashboard
could not show the cap or whether limiting is enabled.
## Changes
- Add `SERVER_CONSUMPTION_QUOTA_UTILIZATION` (global gauge) and emit the
server-level utilization through it instead of the per-table gauge.
- Add `SERVER_CONSUMPTION_RATE_LIMIT` (global) and `CONSUMPTION_RATE_LIMIT`
(per-table) exposing the configured caps on limiter setup/config change (server
gauge is set to `-1` when disabled; server utilization is reset to `0` on
disable so the two server gauges stay consistent).
- Widen the per-minute aggregate and the `emit()` read to `long` to fix the
byte-mode overflow.
## Backward-incompatible metric change
The server-wide quota utilization moves from
`consumptionQuotaUtilization{table="realtimeRowsConsumed"}` to the new global
gauge `serverConsumptionQuotaUtilization`. Dashboards/alerts keyed on the old
series should switch. Please apply the `backward-incompat` label.
## Testing
`RealtimeConsumptionRateManagerTest` — 11/11 pass, covering the
byte-overflow regression (tracker + `emit()`), server-vs-partition gauge
routing, and configured-cap emission on create/update/disable paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
