xiangfu0 opened a new pull request, #19998:
URL: https://github.com/apache/pinot/pull/19998
## Problem and behavior
A sealed segment currently retains a `Schema`, a sorted column-metadata map,
and one `ColumnMetadataImpl` per column for its serving lifetime. This
multiplies heap usage for tables with many wide segments. After local segment
preprocessing, the serving loader now switches to a versioned
`segment_metadata.bin` sidecar. Fixed-size sorted column records and variable
blobs are read through a read-only file mapping; numeric statistics are read
directly, while other values are decoded when used. The canonical
`metadata.properties` remains unchanged for old readers and preprocessing.
The sidecar validates source `metadata.properties` and `index_map`
fingerprints and its payload checksum. A missing, stale, or corrupt sidecar is
rebuilt locally and atomically replaced. The segment directory closes the
mapping after its readers. Realtime and mutable preprocessing metadata keep
their existing behavior. APIs that explicitly request a full schema or column
map still get one constructed on demand; the serving path uses direct column
lookup.
No configuration change is needed for local sealed segments. For standalone
metadata consumers that own the lifecycle:
```java
try (SegmentMetadataImpl metadata =
SegmentMetadataImpl.loadMapped(segmentDir)) {
ColumnMetadata column = metadata.getColumnMetadataFor("myColumn");
// Read column statistics or index sizes, then close the mapping.
}
```
## Validation
Focused segment metadata, loader, empty-segment, pruning, reader,
virtual-column, and server REST tests passed (139 tests total). Spotless,
Checkstyle, and license checks passed for `pinot-segment-spi`,
`pinot-segment-local`, and `pinot-server`. Compatibility probes covered
BYTES/BIG_DECIMAL bounds, zero-document segments, and metadata JSON.
In a rooted synthetic probe of 128 segments with 1,000 INT columns each,
live heap after GC fell from 37,490,568 to 15,140,040 bytes for the metadata
graph (59.6% less). This is not a full-server heap or query-latency
measurement. The normal loader still parses mutable metadata before switching
to the mapping, so this does not establish a startup-time improvement.
## Rollout consideration
Each served segment adds one memory mapping and a sidecar file. Before
enabling this build on a high-segment-count server, check
`/proc/<java-pid>/maps` against the node's `vm.max_map_count` and leave
headroom for existing index mappings. Production query latency and full-server
memory remain to be measured.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
