shuturmurgh opened a new issue, #19853:
URL: https://github.com/apache/pinot/issues/19853
Add a controller endpoint that returns the **loaded CRC of every segment on
every hosting server** (all replicas), so offline ingestion/validation jobs can
confirm data consistency after segment upload/refresh **without requiring
direct access to server admin APIs**.
## Motivation
Offline push jobs often re-upload (refresh) segments that already exist in
Ideal State. Controllers update ZK metadata and send `REFRESH_SEGMENT`
messages, but that path is fire-and-forget: there is no built-in completion
signal that every replica has loaded the new segment.
For an offline job validation step we need to assert that every replica of
every pushed segment is serving the same CRC as ZK (the newly uploaded segment).
Today that is hard when only the controller is reachable:
| Existing API | Limitation for this use case |
| --- | --- |
| `GET /segments/{table}/crc` | ZK/desired CRC only — not what servers
loaded |
| `GET /segments/{table}/metadata` | Collapses by segment name — loses
per-replica visibility |
| Server `GET /tables/{table}/segments/crc` | Correct data, but server admin
APIs are often not exposed |
Without per-replica loaded CRCs via the controller, validation cannot
reliably prove cluster-wide consistency after refresh.
## Proposed API
```http
GET /segments/{tableNameWithType}/serverCrcs
```
**Response** (External View–style: segment → server → CRC; preserve all
replicas):
```json
{
"seg_A": {
"Server_host1_8098": "crc_A1",
"Server_host2_8098": "crc_A2"
},
"seg_B": {
"Server_host1_8098": "crc_B1",
"Server_host3_8098": "crc_B3"
}
}
```
This matches the shape of the External View API (`segment → { server → state
}`), but with loaded CRC instead of Helix state.
Implementation can mirror existing controller fan-out (`isStale` / segment
metadata):
1. Resolve hosting servers for the table
2. Call each server’s `/tables/{tableNameWithType}/segments/crc`
3. Transpose into `Map<segment, Map<server, crc>>`
Optional: support a `segments` query parameter to limit the response to
segments touched by the job.
> **Naming note:** Existing `GET /segments/{table}/crc` returns ZK CRCs. The
new API should be clearly named/documented as **server-loaded** CRCs (e.g.
`serverCrcs`) to avoid confusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
