navina commented on code in PR #9977:
URL: https://github.com/apache/pinot/pull/9977#discussion_r1051081259
##########
pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerGauge.java:
##########
@@ -45,7 +45,10 @@ public enum ServerGauge implements AbstractMetrics.Gauge {
// Dedup metrics
DEDUP_PRIMARY_KEYS_COUNT("dedupPrimaryKeysCount", false),
CONSUMPTION_QUOTA_UTILIZATION("ratio", false),
- JVM_HEAP_USED_BYTES("bytes", true);
+ JVM_HEAP_USED_BYTES("bytes", true),
+ // Consumer lag metrics
+ LLC_RECORDS_LAG("recordCount", false),
+ LLC_AVAILABILITY_LAG_MS("availabilityLagMs", false);
Review Comment:
> Our SREs complained about the per metrics approach as it would make their
job harder, they wanted a single point metric that tells them the worst case
scenario.
this is a good point. But we and some users have often found the need to
look at per-partition level metrics during debugging. This is esp. true for
non-kafka sources for which pinot provides no equivalent lag metrics.
> Ack on this having removed the support for offset, but I think that was
important.
In most practical scenarios, offset difference doesn't tell you much. But it
can be useful during debugging and this is definitely one of the widely
asked/popular metric. However, computing this in Pinot layer is quite expensive
as it requires sending a request to the source kafka broker each time. Instead,
the kafka consumer client provides APIs to fetch this metric that it holds
internally. I think going forward, we should leverage that API or directly try
to export the kafka consumer client's mbean which contains the lag.
Moreover, the offset lag doesn't apply to non-kafka sources anyway.
> When we do try to consume and no events are found we display 0 delay as if
there were events we will consume them with no delay
That's not what this PR is doing though. When there are no events, the delay
remains the same as the last sample. This along with the consumption rate
should tell you whether there were any new events or not.
> Please let me know if the meeting I set for next week suits your schedule
and if not let me know what time will be best.
I don't see an invite in any of my emails. I can DM you my email in the OSS
slack.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
