KKcorps commented on code in PR #19632:
URL: https://github.com/apache/pinot/pull/19632#discussion_r4079165397
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java:
##########
@@ -1489,6 +1489,26 @@ private void releaseConsumerSemaphore() {
}
}
+ /// Updates the [ServerGauge#LLC_PARTITION_CONSUMING] gauge when this
consuming segment stops.
+ ///
+ /// When the partition is retired -- a stream shard that has been
split/merged and fully consumed, i.e.
+ /// [#_endOfPartitionGroup] is set -- no successor consuming segment is ever
created for it, so leaving the
+ /// gauge at 0 would make it linger at 0 forever and raise a false-positive
RealtimeIngestionStopped alert.
+ /// In that case the gauge is removed so the series goes absent. For every
other stop (a normal commit that
+ /// gets a successor consuming segment, or a hold) the gauge is set to 0 as
before.
+ private void markConsumingStopped() {
+ if (_endOfPartitionGroup) {
+ _serverMetrics.removeTableGauge(_clientId,
ServerGauge.LLC_PARTITION_CONSUMING);
Review Comment:
Could we keep this at `0` until the final segment commits successfully?
If the build fails after EOF, we enter `ERROR` but remove the gauge here,
which hides a real ingestion stall.
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java:
##########
@@ -1725,7 +1745,7 @@ public void goOnlineFromConsuming(SegmentZKMetadata
segmentZKMetadata)
} catch (Exception e) {
Utils.rethrowException(e);
} finally {
- _serverMetrics.setValueOfTableGauge(_clientId,
ServerGauge.LLC_PARTITION_CONSUMING, 0);
+ markConsumingStopped();
Review Comment:
Could we check that this segment still owns the gauge before removing it?
An EOF replica can lose to an earlier committed offset, then this `finally`
can delete its successor's gauge after the successor starts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
