xiangfu0 opened a new pull request, #19477:
URL: https://github.com/apache/pinot/pull/19477
## What
Two changes that shrink what a segment holds per column.
- `ColumnMetadataImpl` layout: the six booleans and the forward-index
encoding pack into one flags field, and the references only a partitioned
column or an OPEN_STRUCT parent/child carries (partition function and
partitions, parent column, sparse keys) move into a lazily allocated holder
that stays null for every ordinary column. Compression stats stay a direct
reference because the creator writes them for every raw column. The object goes
from 96 to 72 bytes. None of it is visible through the getters, so the REST
payload is unchanged.
- The per-segment `Schema` is no longer built during `init()`. It is derived
from the column metadata on the first `getSchema()` and cached, and invalidated
by `removeColumn`. `ImmutableSegmentLoader` builds the built-in virtual column
specs directly, so a normal segment load never materializes it. A test-visible
counter plus a loader assertion keep it that way in CI.
## Tests
`SegmentMetadataImplTest`: the lazily built schema equals the eagerly built
one, includes the built-in virtual columns after load, is invalidated by
`removeColumn`, and is still unbuilt after `ImmutableSegmentLoader.load`.
`ColumnMetadataImplTest` covers the packed flags and the holder fields for
partitioned and OPEN_STRUCT columns.
## Why
A server keeps one metadata object graph per (segment, column) for as long
as the segment is loaded, so on wide tables the per-column footprint decides
how many segments a server can hold. Measured end to end on a 1000-column
segment, this series takes the heap retained at load from **4.08 MB to 0.175 MB
per segment** (4,080 to 174 bytes per column), with a fully compacting
collector on both sides. No on-disk format change, the
`/tables/{table}/segments/{segment}/metadata` JSON stays byte-identical, and
every public and SPI signature keeps working.
## Stack
Part 7 of 9, based on `xiangfu0/data-3221-5-fieldspec-interner`. Review only
this part's own commits; the rest of the diff belongs to the parts below it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
