xiangfu0 opened a new pull request, #19486: URL: https://github.com/apache/pinot/pull/19486
## What Two things the segment preprocess does on every load asked the segment metadata for its `Schema`. The schema is derived and then cached, so each one pinned a `Schema` per loaded segment for the segment's whole life: - `ForwardIndexHandler#computeOperations` wants the set of physical column names. - `ColumnMinMaxValueGenerator` wants the columns its mode selects — the default mode is `ALL`, so this runs on every load too. Both questions are answered by the column metadata the schema is itself derived from. `SegmentMetadata#getPhysicalColumnNames()` walks the column metadata (falling back to the schema for a segment that holds no column metadata, i.e. a CONSUMING one), and the min/max generator selects straight off each column's field spec instead of off `schema.getAllFieldSpecs()`. No behavior change: the same columns are selected, in the same order the column metadata is held in, and `getSchema()` still returns the same schema for anyone who asks for it. ## Why Found on a production server holding 13.6k loaded segments: one `Schema` per segment, ~144 MB of tree entries and list slots, all of it a second copy of what the column metadata already holds. Part 7 of this series (#19478) made the per-segment schema lazy; these two callers were the reason it was still built for every segment anyway. ## Tests `SegmentMetadataImplTest#testPreprocessDoesNotBuildTheSegmentSchema`: `getPhysicalColumnNames()` returns what `getSchema().getPhysicalColumnNames()` returns without materializing a schema, and a full `SegmentPreProcessor#process()` over a v3 segment materializes none. The test fails if either call site is reverted. ## Stack Part 10, based on #19481. Review only this part's own commit; the earlier parts account for the rest of the diff. 1. #19480 lazy index-size storage 2. #19473 canonical default-null values and interned per-column strings 3. #19474 delegating immutable DataSourceMetadata 4. #19475 presence-mask index container 5. #19476 weak FieldSpec interner 6. #19477 opt-in lazy column materialization 7. #19478 slim ColumnMetadataImpl and lazy per-segment Schema 8. #19479 primitive numeric min/max 9. #19481 sorted-array column metadata store 10. this PR — stop the preprocess from building a schema per segment 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
