xiangfu0 opened a new pull request, #19475:
URL: https://github.com/apache/pinot/pull/19475
## What
Every segment of a table parses its own `FieldSpec` per column, and after
part 2 those specs are value-identical across segments. `ColumnMetadataImpl`
now interns them through a weak interner keyed by
`FieldSpec.equals`/`hashCode`, so all segments of a table share one instance
per distinct column definition and schema evolution still yields a distinct
instance per version. They are held weakly, so unloading the last segment
releases them.
`ComplexFieldSpec` parents are not interned (no value equality); their
children are.
Segment-derived specs must therefore be treated as immutable, which is
documented on `ColumnMetadata#getFieldSpec()` and
`SegmentMetadata#getSchema()`. An audit of all main sources found no caller
that mutates one: the virtual-column provider mutates only specs it builds
itself.
## Tests
`ColumnMetadataImplTest`: equal configurations parsed twice share one
instance; differing default value, max length, data type, single-value or
format do not alias; COMPLEX parents are not interned while children are; an
unreferenced spec is released after GC.
## Why
A server keeps one metadata object graph per (segment, column) for as long
as the segment is loaded, so on wide tables the per-column footprint decides
how many segments a server can hold. Measured end to end on a 1000-column
segment, this series takes the heap retained at load from **4.08 MB to 0.175 MB
per segment** (4,080 to 174 bytes per column), with a fully compacting
collector on both sides. No on-disk format change, the
`/tables/{table}/segments/{segment}/metadata` JSON stays byte-identical, and
every public and SPI signature keeps working.
## Stack
Part 5 of 9, based on `xiangfu0/data-3221-3-datasource-adapter`. Review only
this part's own commits; the rest of the diff belongs to the parts below it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
