Jackie-Jiang opened a new pull request, #19439: URL: https://github.com/apache/pinot/pull/19439
## Summary `NullDataSource` is now a single class in the `datasource` package of `pinot-segment-local`, shared by MAP and OPEN_STRUCT, and built exactly like a default column (a schema column missing from the segment): it reuses `DefaultNullValueVirtualColumnProvider` and only adds an all-null `NullValueVectorReader`. Every absent key therefore gets a single-entry dictionary holding the field's default null value, a `ConstantSortedIndexReader` (or the constant MV forward and inverted readers), and `ColumnMetadataImpl` with cardinality 1, sorted for single-value, and the default as min/max, so with null handling on the key reads as NULL and with it off as the default. Previously MAP and OPEN_STRUCT each had their own all-null source. The MAP one was hardcoded as an `INT` column with `numDocs = 0`, so an absent key's type disagreed with the map's value type, and both served hardcoded dimension defaults from a raw forward index, ignoring `BOOLEAN` / `TIMESTAMP` defaults (`0`, not `MIN_VALUE`), METRIC defaults and custom `defaultNullValue`s. ### Absent keys are resolved by the data sources `OpenStructDataSource.getDataSource(key)` and `MapDataSource.getDataSource(key)` never return `null` anymore: a definitively absent key resolves to the `NullDataSource` inside the implementation. The new default method `OpenStructDataSource.getKeyFieldSpec(key)` holds the rule the contract already documented (declared child spec, else a single-value STRING), and `ImmutableOpenStructDataSource` uses it for sparse keys too. Both SPIs are implemented only in this repo. - `ProjectionBlock`, `ItemTransformFunction` and `MapFilterOperator` lose their "synthesize when `null`" fallbacks. `MapFilterOperator.buildAbsentKeyFilterOperator` and its raw-value `matchesDefaultNullValue` switch are gone: the one-entry dictionary makes every EQ / NOT_EQ / IN / NOT_IN / RANGE evaluator fold to always-true or always-false, and `getLeafFilterOperator` already handles always-true under null handling through the null vector, so the regular per-key path yields the same match-all / match-none operators. - `BaseMapDataSource.getDataSource` no longer catches, logs and swallows exceptions while building a key's data source, which turned a broken per-key index into "absent key"; it propagates them. - `ItemTransformFunction` uses the per-key null vector for an absent MAP key, which is exact, and keeps falling back to the map column's bitmap for present keys, which carry no per-key null information. ### Also `BaseConstantValueVirtualColumnProvider.buildMetadata` now sets the total number of entries to the doc count, so `getNumValues()` is correct for every constant-value virtual column instead of `0`. ### Behavior notes - Absent OPEN_STRUCT and MAP keys are dictionary-encoded like default columns, so `item()` over them now reports a dictionary. - A key excluded by the sparse manifest on a segment that also has a sparse JSON index goes through `JsonMatchFilterOperator` instead of the per-key fold: same result, different EXPLAIN. - No `MapIndexReader` in this repo returns `null` from `getIndexes`, so the MAP absent-key path is only reached through the contract branch; the type fix matters for implementations that do. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
