Jackie-Jiang opened a new pull request, #19367: URL: https://github.com/apache/pinot/pull/19367
## Summary Adds a second star-tree variant that pre-aggregates with null-handling-on semantics, selected with a new `nullHandlingEnabled` flag on `StarTreeIndexConfig`. Today a star-tree folds nulls into the column's default value and counts them, which is right for a null-handling-off query and wrong for a null-handling-on one — so a query that enables null handling can only use a star-tree when nothing it touches is actually null, and otherwise drops to a raw scan. Both variants can live in the same segment. The flag is part of the builder config's identity, so they survive dedup and one is never reused for the other. Depends on #19218: excluding nulls from the pre-aggregation only agrees with a raw scan once the aggregation functions themselves skip null rows, which is now true of all of them. ### Dimensions Nulls are stored under a reserved dictionary id — the column's cardinality, one past the last real id — so null rows form their own tree node instead of collapsing into the default value's node. The forward index is widened to `cardinality + 1` values and `StarTreeLoaderUtils` recomputes the bit width to match. A null vector is written for dimensions as well as metrics: the reserved id is out of range for the segment dictionary the star-tree shares, so the query side has to recognise a null from the vector rather than from the stored id. ### Metrics Null values are excluded from the pre-aggregation. A group left with no non-null input has no aggregated value to store, so the metric keeps a placeholder in the forward index and its nullness lives in the metric's null vector — the same shape a regular column is stored in. `COUNT` is the one aggregator that still answers for such a group itself, through `ValueAggregator#getAllNullAggregatedValue`. It is read back by summing the pre-aggregated column rather than through the null vector, and `0` is exactly the placeholder a null would leave behind, so recording the group in the vector would cost a bit per group and buy nothing. Every other aggregator takes the `null` default, which keeps an all-null group down to a placeholder plus one null-vector bit rather than a serialized empty sketch, and stops `getMaxAggregatedValueByteSize` sizing the column for one. `COUNT(column)` becomes representable as a stored function-column pair so a null-aware tree can pre-aggregate per-column non-null counts. Config parsing keeps the previous `COUNT__col` → `count__*` normalisation for regular trees, so existing table configs are unaffected. ### The off-heap builder The on-heap builder holds `Record`s in memory, where a null metric needs no encoding. The off-heap builder serializes every record to a temporary store and could not express one — it dereferenced the missing value while serializing. Nullness is now held in a per-metric `RoaringBitmap` keyed by doc id, and the serialized record carries the aggregated type's zero in its place. Keying on the doc id is safe because records never move: sorting permutes an array of doc ids and compares through it, leaving each record where it was written. The record layout is unchanged, so `FixedSizeRecordOffsets`' arithmetic still holds. ### Query routing Each query is routed to a tree built in the matching mode. A null-handling-on query may still fall back to a regular tree when none of the columns it touches contains a null value — that is the check that already existed for exactly this case, now extracted so both paths share it. The resolved function-column pairs travel with the project operator through `AggregationInfo`, because `COUNT` resolves differently per tree and the executors must read back the same columns that were projected. `AggregationInfo` derives `isUseStarTree()` from whether those pairs are present rather than carrying a separate flag, so the two cannot disagree. ### Compatibility A deployment that does not opt in is unaffected: an old segment has no `null.handling.enabled` property and reads `false`, an old table config defaults the flag to `false`, and the new mode filter skips nothing when both are `false`. Same trees built, same tree chosen, same answers. The behavioural change is confined to segments that opt into a null-aware tree, where a null-handling-on query now gets the same answer a raw scan would give instead of falling back to one. ### Tests `NullAwareStarTreeBuilderTest` builds a segment whose metric is null for a whole dimension group and covers both that the all-null group survives the build and that the two variants coexist and disagree — a regular tree reports the column default as its minimum and has no null vector, a null-aware one excludes those rows and does. The build mode is a `@DataProvider` parameter rather than a random choice. `BaseStarTreeV2Test` picks between `ON_HEAP` and `OFF_HEAP` with `RANDOM.nextBoolean()` across all its subclasses, which would have made the off-heap gap above a coin-flip flake rather than a failure. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
