KKcorps opened a new pull request, #19637:
URL: https://github.com/apache/pinot/pull/19637
Replica differences in upsert results are hard to investigate from
valid-document counts alone: two replicas can report the same count while
selecting different rows, and a count mismatch can reflect different ingestion
progress or persisted snapshot files.
This adds opt-in diagnostics to the **server** `POST
/tables/{tableNameWithType}/validDocIdsMetadata` endpoint. Pass
`includeDiagnostics=true` and query the selected segments on each replica. The
existing counts, segment CRCs, bitmap type and server identity remain the
context for comparison.
### How the fields help triage
The additional fields are grouped under `diagnostics` in each segment entry:
| Field | Question it helps answer |
| --- | --- |
| `validDocIdsCrc32` | Do equal valid-document counts hide different
document IDs? Compare it only for the same segment contents and bitmap type. A
mismatch identifies a segment whose bitmap membership needs inspection. |
| `captureStartTimeMs`, `captureEndTimeMs` | When did this server collect
the observation, and how long did collection span? Use the window to correlate
differences with ingestion, restart, replacement and cleanup logs. These use
the server clock. |
| `snapshotFileAgeMs` | Are we comparing persisted files with different
write ages? This is the age of the selected local snapshot file at capture
completion. It can direct investigation toward snapshot/recovery history, but
an old unchanged file can still be valid. Omitted for in-memory reads or when
file identity/mtime cannot be established across the read. |
| `consumingSegmentOffsetsBefore`, `consumingSegmentOffsetsAfter` | Was an
observed consumer advancing during collection, or were the replicas at
different consumption positions? The maps preserve the consumer segment name
and its next offset to consume, including overlapping consumers in one
partition. |
The offsets are current observations around the API bitmap read, **not the
offsets at which a persisted snapshot was created**. They are included for
segments with an LLC partition identity; empty maps mean no usable offsets were
observed, and absent maps mean the partition could not be identified. Only
consumers acquired for this request are observed; matching offsets do not rule
out concurrent lifecycle work or an in-progress record update.
### Manual use
```bash
curl -sS -X POST \
"$SERVER/tables/T_REALTIME/validDocIdsMetadata?validDocIdsType=IN_MEMORY&includeDiagnostics=true"
\
-H 'Content-Type: application/json' \
-d '{"segments":["SEGMENT_NAME"]}'
```
Query each replica's server admin endpoint. Compare the same segment
contents and bitmap type; use `SNAPSHOT` for persisted state and the existing
`*_WITH_DELETE` modes for queryable IDs. Download mismatched bitmaps through
`validDocIdsBitmap` for further inspection.
The controller aggregate endpoint currently retains one response per
segment. This change stays on the server endpoint.
### Semantics and cost
- Count and CRC use the same detached bitmap. CRC32 hashes ascending IDs as
four big-endian bytes, independent of Roaring representation. Different CRCs
identify different sampled memberships; equal CRCs can collide and say nothing
about row payloads or a common replica boundary.
- File age is omitted when identity/size/mtime changes across the read, file
identity is unavailable, or mtime is invalid/future. It is not time since the
last successful partition snapshot attempt.
- Default behavior is unchanged. Opting in scans selected bitmap IDs and
reads local file attributes/consumer offsets. There is no per-record ingestion
work, stream RPC, forced snapshot, or ingestion lock.
### Validation
- 14 targeted tests passed: bitmap membership versus cardinality,
representation-independent CRCs, fixed encoding, capture windows,
same-partition/overlapping consumers, unknown offsets, selected snapshot file
age, replacement/missing/future files, all four HTTP bitmap modes, and
unchanged default responses.
- Server-module Spotless, license formatting/check and Checkstyle passed.
- Server-module `test-compile` with `-Dmaven.compiler.showDeprecation=true
-Dmaven.compiler.showWarnings=true` passed with no warnings on added lines.
Dependencies were first built and installed from this checkout. The broader
warning-enabled reactor attempt hit the existing
`ZstandardDecompressor.java:51` missing `org.jetbrains.annotations.NotNull`
dependency; the standard reactor build and targeted tests passed.
Targeted test command:
```bash
./mvnw -pl pinot-server -am \
-Dtest='ValidDocIdsMetadataDiagnosticsTest,TablesResourceTest#testValidDocIdsMetadata*'
\
-Dsurefire.failIfNoSpecifiedTests=false \
-Dcheckstyle.skip -Dspotless.check.skip -Dlicense.skip test
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
