xiangfu0 opened a new pull request, #18731: URL: https://github.com/apache/pinot/pull/18731
## Problem In strict replica-group routing (`StrictReplicaGroupInstanceSelector`, and `ReplicaGroupInstanceSelector` for upsert/dedup tables), when an old segment is missing (not ONLINE/CONSUMING in ExternalView) on an instance, that instance is marked unavailable **for every segment sharing the same ideal state instance set**. This is correct when the segment is available on some other instance in the group: it forces the whole group to route to an instance that has all the segments. However, when a segment is not online on **any** instance in its instance set, this logic marks **all** instances of the group unavailable, taking down every other segment hosted by the same instances — even though excluding instances cannot make the missing segment servable and provides no consistency benefit. This caused a production incident: during tier relocation while one cold-tier replica group was restarting, a `bestEfforts` rebalance advanced the IdealState past ExternalView convergence, leaving a handful of relocating segments ONLINE nowhere for ~1 minute. Those few segments poisoned both cold instances of every instance group, and the broker reported the **entire cold tier (~29.5k segments) unavailable**: ``` 29562 segments unavailable, sampling 10: [...], with routing policy: strictReplicaGroup [realtime] ``` The same mechanism also lets a single corrupt segment (ERROR on all replicas) black out a whole replica-group set. ## Fix In `updateSegmentMapsForUpsertTable`, an old segment with no online instance within its ideal state instance set no longer contributes to the unavailable-instance computation for its group. It gets an empty candidate list and is reported as unavailable **individually**, while all other segments on those instances keep serving. The strict replica-group guarantee is unchanged for any segment that is available on at least one instance: instances missing such a segment are still excluded for the whole group, so all served segments of a group are still served from a consistent instance (verified by the last phase of the new regression test). ## Behavior change Before: a segment unavailable everywhere caused **all** segments sharing its instance set to be reported unavailable, and the query served no rows from any of them. After: only the truly unservable segment is reported unavailable; the rest of the group keeps serving. This applies to `StrictReplicaGroupInstanceSelector` (all tables) and `ReplicaGroupInstanceSelector` (upsert/dedup tables), since both share `updateSegmentMapsForUpsertTable`. For upsert tables this means a query can now return rows from the remaining segments of a partition while the nowhere-online segment is reported unavailable; records whose latest version lives in that segment may surface earlier versions. Note that in both the old and the new behavior the query response carries a `BROKER_SEGMENT_UNAVAILABLE` exception, so clients that treat partial results as failures are unaffected. Given that the previous behavior also returned partial (more incomplete) results with the same exception, this is shipped without a config gate. ## Testing - New regression test `testStrictReplicaGroupSegmentNotOnlineAnywhereDoesNotMarkGroupDown` covering: (1) the incident scenario (relocated segment not online anywhere does not take down the group), (2) recovery once the segment loads, and (3) the preserved group-down behavior for a segment that is online on one instance but missing on another. - Updated existing test expectations that codified the old blackout behavior (each with explanatory comments). - Full `pinot-broker` module test suite passes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
