xiangfu0 opened a new pull request, #19473:
URL: https://github.com/apache/pinot/pull/19473
## What
`ImmutableDataSource.ImmutableDataSourceMetadata` copied ten fields out of
`ColumnMetadata` into a per-column snapshot, about 56 bytes each. It now holds
a single `ColumnMetadata` reference and delegates, which keeps the semantics
identical because `ColumnMetadata` is immutable after construction:
`getNumDocs` -> `getTotalDocs`, `getNumValues` -> `getTotalNumberOfEntries`,
and `getMaxNumValuesPerMVEntry` preserves the `-1` contract for single-value
columns. `ImmutableMapDataSource` gets the same treatment, keeping its
unconditional `isSorted() == false` and its `getMaxRowLengthInBytes` override.
`DataSourceMetadata` is never serialized and no public signature changes.
## Tests
New `ImmutableDataSourceTest` and `ImmutableMapDataSourceTest`: every
`DataSourceMetadata` getter equals the corresponding `ColumnMetadata` getter
for a single-value INT column with min/max and partitions, a multi-value STRING
column, and a virtual-column-style spec.
## Why
A server keeps one metadata object graph per (segment, column) for as long
as the segment is loaded, so on wide tables the per-column footprint decides
how many segments a server can hold. Measured end to end on a 1000-column
segment, this series takes the heap retained at load from **4.08 MB to 0.175 MB
per segment** (4,080 to 174 bytes per column), with a fully compacting
collector on both sides. No on-disk format change, the
`/tables/{table}/segments/{segment}/metadata` JSON stays byte-identical, and
every public and SPI signature keeps working.
## Stack
Part 3 of 9, based on `xiangfu0/data-3221-1-lazy-index-sizes`. Review only
this part's own commits; the rest of the diff belongs to the parts below it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
