soumitra-st opened a new pull request, #12957:
URL: https://github.com/apache/pinot/pull/12957
# Problem
In multiple Pinot deployments, we have seen the REALTIME_INGESTION_DELAY_MS
and END_TO_END_REALTIME_INGESTION_DELAY_MS metrics monotonically grow even
though there were no active Kafka consumers.
## How did I figure out that there are no active Kafka consumers?
consumingSegmentsInfo API returned the below response:
```
{
"_segmentToConsumingInfoMap": {
"telemetry__0__2399__20240415T1650Z": [],
"telemetry__1__2435__20240415T1600Z": [],
"telemetry__2__757__20240415T1632Z": [],
"telemetry__3__2413__20240415T1642Z": []
}
}
```
The above response shows that consumers are not connected. I also checked
the thread dump, and there are no Kafka consumer threads.
## Image showing REALTIME_INGESTION_DELAY_MS grow monotonically
<img width="1245" alt="zoomed-out-ingestion-delay"
src="https://github.com/apache/pinot/assets/127247229/32ca9545-bb55-408d-abbb-fc4680817e52";>
## Zoomed out image showing REALTIME_INGESTION_DELAY_MS grow monotonically
every minute
<img width="1279" alt="zoomed-in-ingestion-delay"
src="https://github.com/apache/pinot/assets/127247229/dc6d5eb8-2b17-4e79-a028-72268526561d";>
# Solution
Changed the supplier to provide constant value to publish
REALTIME_INGESTION_DELAY_MS and END_TO_END_REALTIME_INGESTION_DELAY_MS metrics.
`bugfix`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
