xiangfu0 opened a new pull request, #19476:
URL: https://github.com/apache/pinot/pull/19476
## What
Opt-in, default off:
`pinot.server.instance.segment.lazy.column.materialization`, read through
`InstanceDataManagerConfig#isLazyColumnMaterialization()` and carried on
`IndexLoadingConfig`.
When enabled, `ImmutableSegmentLoader.load(SegmentDirectory, ...)` no longer
builds a `PhysicalColumnIndexContainer` and an `ImmutableDataSource` for every
column at load. It builds a `ColumnMaterializer` holding the segment reader and
an immutable snapshot of the field index configs, and `ImmutableSegmentImpl`
creates the container and data source on the first `getDataSource(column)`
through `computeIfAbsent`, so creation is single flight and a failure leaves no
mapping. Virtual columns, star-tree containers, open-struct children and the
multi-column text index keep their eager path, and `destroy()` closes only what
was materialized.
The trade-off, stated plainly: an unreadable index on a never-queried column
surfaces on first access instead of at segment load, and the `hasIndexFor`
probing of 13 index types per column moves with it. That is why the flag
defaults to off.
## Tests
`ImmutableSegmentImplTest` in lazy mode: nothing is created at construction;
N concurrent callers produce exactly one container; `destroy()` closes only
materialized containers and later materialization throws; `getIndex(column,
type)` materializes; with the flag off the behaviour is byte-identical to today.
## Why
A server keeps one metadata object graph per (segment, column) for as long
as the segment is loaded, so on wide tables the per-column footprint decides
how many segments a server can hold. Measured end to end on a 1000-column
segment, this series takes the heap retained at load from **4.08 MB to 0.175 MB
per segment** (4,080 to 174 bytes per column), with a fully compacting
collector on both sides. No on-disk format change, the
`/tables/{table}/segments/{segment}/metadata` JSON stays byte-identical, and
every public and SPI signature keeps working.
## Stack
Part 6 of 9, based on `xiangfu0/data-3221-4-index-container-mask`. Review
only this part's own commits; the rest of the diff belongs to the parts below
it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
