kpatel006 opened a new pull request, #19891:
URL: https://github.com/apache/pinot/pull/19891
## Background: Panic Routing
Panic routing is when the broker finds no available server for one or more
segments in a query. Today the broker gives up on those segments: it marks them
unavailable and returns partial results without them.
Fail-open routing handles this differently. If a segment still has candidate
servers, but they're all disabled or marked unhealthy, the broker sends the
request to one of them anyway. A disabled server often still has the segment
loaded and can answer, so serving from it is better than dropping the segment
from the result.
This PR doesn't turn on fail-open routing. It adds metrics that show how
often panic routing happens and how much of it fail-open could fix.
## Summary
Adds two broker meters that show how often fail-open routing would help,
without changing any routing behavior.
The change splits unavailable segments into two kinds:
| Kind | Meaning | Can fail-open help? |
|---|---|---|
| **Unavailable, with candidates** | The segment has replicas, but every
candidate instance is disabled | **Yes**: a disabled instance still holds the
data |
| **Unavailable, without candidates** | No instance is hosting the segment
at all | No: there is nowhere to route |
### Changes
- **`BrokerMeter`**: two new meters:
- `FAIL_OPEN_ELIGIBLE_REQUESTS` (global): counts queries that touch at
least one segment that is unavailable but has candidates.
- `FAIL_OPEN_ELIGIBLE_SEGMENTS` (per table): counts each time a segment
*moves into* that state. Unchanged refreshes and recoveries don't count.
- **`SegmentStates`**: now also stores which unavailable segments have no
candidates.
- **`BaseInstanceSelector`**: records that state during
`refreshSegmentStates()` and updates both meters.
- **`InstanceSelectorTest`**: covers the full lifecycle for every selector
type: no candidates → disabled candidate → unchanged refresh → recovery →
disabled again.
## Motivation
Today every unavailable segment is reported the same way
(`NO_SERVING_HOST_FOR_SEGMENT`), so you can't tell which ones fail-open could
actually serve. These meters show how much traffic fail-open would affect, and
which tables, before we turn it on.
## Diagram
```mermaid
flowchart TD
subgraph Refresh["refreshSegmentStates()"]
A[Old segment] --> B{Any enabled<br/>candidates?}
B -- yes --> C[Routable]
B -- no --> D{Any candidates<br/>at all?}
D -- no --> E[Unavailable, without candidates]
D -- yes --> F[Unavailable, with candidates]
F --> G{Already in this<br/>state before?}
G -- no --> H[["FAIL_OPEN_ELIGIBLE_SEGMENTS += 1"]]
G -- yes --> I[No-op]
end
subgraph Select["select() — per query"]
Q[Query segments] --> R{Touches a segment that's<br/>unavailable but
has candidates?}
R -- yes --> S[["FAIL_OPEN_ELIGIBLE_REQUESTS += 1"]]
R -- no --> T[No-op]
end
Refresh -.->|SegmentStates| Select
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
