xiangfu0 opened a new pull request, #19479:
URL: https://github.com/apache/pinot/pull/19479
## What
`SegmentMetadataImpl` kept its columns in a `TreeMap`, about 40 bytes of
red-black-tree node per column. They now live in sorted parallel arrays with
binary-search lookup, published as one immutable holder behind a single
volatile reference so a reader can never pair new names with old metadata, and
replaced copy-on-write.
`getColumnMetadataMap()` still works: the map view is derived on demand and
cached, and the load and query paths use the array accessors instead, so a
normal segment load never materializes it. A test-visible counter plus a loader
assertion keep it that way. Additive accessors were added for the callers that
only need a count, a lookup or an iteration, and virtual columns are added
through an explicit `addColumnMetadata` rather than by mutating the returned
map.
Column order, `getAllColumns()`, `getPhysicalColumnNames()` and `toJson` are
unchanged.
## Tests
`SegmentMetadataImplTest`: the derived map equals the previous one and stays
unbuilt after `ImmutableSegmentLoader.load`; adding and removing columns keeps
the arrays sorted and consistent; consuming segments behave consistently across
all accessors. New `SortedStringArraySetTest` covers the sorted-set view
including range bounds.
## Why
A server keeps one metadata object graph per (segment, column) for as long
as the segment is loaded, so on wide tables the per-column footprint decides
how many segments a server can hold. Measured end to end on a 1000-column
segment, this series takes the heap retained at load from **4.08 MB to 0.175 MB
per segment** (4,080 to 174 bytes per column), with a fully compacting
collector on both sides. No on-disk format change, the
`/tables/{table}/segments/{segment}/metadata` JSON stays byte-identical, and
every public and SPI signature keeps working.
## Stack
Part 9 of 9, based on `xiangfu0/data-3221-7-slim-metadata-lazy-schema`.
Review only this part's own commits; the rest of the diff belongs to the parts
below it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
