KKcorps opened a new pull request, #19730: URL: https://github.com/apache/pinot/pull/19730
When a stream partition ends, like a Kinesis shard that is fully read after a split or merge, Pinot never creates a new consuming segment for it. The `llcPartitionConsuming` gauge then stays at 0 on every replica, and alerts on that gauge fire forever. This covers the same problem as #19632, but it also handles replicas that caught up or downloaded the last segment, and it doesn't rely on a replica seeing the end locally. The controller now emits a per-partition `partitionEndOfLife` gauge. It is set to 1 when the controller stops a partition for end of life, either at commit or in `RealtimeSegmentValidationManager`, and to 0 when the validation task creates a consuming segment for it again. It is dropped when the partition has no segments left or the controller stops leading the table. Alerts can then skip partitions where it is 1. On the server, when `IngestionDelayTracker` confirms through the ideal state that the server no longer hosts a CONSUMING segment for a partition, it now also removes that partition's `llcPartitionConsuming` gauge. This works the same way on every replica. Tests: new cases in `PinotLLCRealtimeSegmentManagerTest` (commit, validation, retention, a partition coming back, leader cleanup) and `IngestionDelayTrackerTest`, and both fail when their main code path is removed. `RealtimeSegmentDataManagerTest`, `RealtimeSegmentValidationManagerTest`, the Prometheus export test, spotless, checkstyle and license checks pass. The compile with warning flags stops at the unchanged `ZstandardDecompressor.java:51`, as on master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
