tarun11Mavani opened a new pull request, #19040: URL: https://github.com/apache/pinot/pull/19040
## Summary This is **PR 3/4** in the OPEN_STRUCT stack — the query engine changes that make `DataType.OPEN_STRUCT` columns queryable. Without it, declaring OPEN_STRUCT in a schema and ingesting data works (PR 2), but all queries crash or fall to per-doc expression scan. RFC: https://docs.google.com/document/d/14kPmjDTKbO8l0ql4rrN7I5Yki5pqMw6GeGmxxc9grsU/edit?tab=t.0 PR 1 (SPI + data model): #18368 — merged PR 2a (SPI utilities, index registration): #18710 — merged PR 2b (storage layer): #18643 — merged ### What this PR adds **Projection path (`SELECT myStruct['key']`):** - `ItemTransformFunction` — relaxes `instanceof MapDataSource` to also accept `OpenStructDataSource`. When `getDataSource(key)` returns null (absent key in mutable or fully-materialized segment), wraps in `NullDataSource`. - `ProjectionBlock` — same pattern for path-based `[column, key]` value set access. **Filter path (`WHERE myStruct['key'] = 'val'`):** - `MapFilterOperator` — refactored from 2-way dispatch to 3-way: 1. **PER_KEY_INDEX** (new) — for `OpenStructDataSource` columns. Uses `isMaterialized(key)` / `isFullyMaterialized()` for sub-dispatch: - Materialized key → rewrite predicate LHS, build `PredicateEvaluator`, delegate to `FilterOperatorUtils.getLeafFilterOperator` (picks inverted/range/sorted/scan automatically) - IS_NULL/IS_NOT_NULL → read `NullValueVectorReader` bitmap directly → `BitmapBasedFilterOperator` - Absent key (fully materialized) → IS_NULL → `MatchAllFilterOperator`, else → `EmptyFilterOperator` - Sparse key (not fully materialized) → fall through to expression scan 2. **JSON_MATCH** (existing, unchanged) — MAP columns with JSON index 3. **EXPRESSION_FILTER** (existing, unchanged) — fallback scan - Single `_delegate` field replaces the old dual-field pattern. EXPLAIN output reports `delegateTo:per_key_index|json_match|expression_filter`. **Aggregation path (`SELECT COUNT(DISTINCT myStruct['key'])`):** - `AggregationPlanNode` — `resolveDataSource()` + `tryResolveKeyedDataSource()` resolve `item(col, 'key')` FUNCTION expressions to per-key DataSources. Enables dictionary/metadata-based non-scan aggregation (COUNT DISTINCT, MIN, MAX) on dense OPEN_STRUCT keys. Sparse keys return null → scan path. ### Supported predicate types on per-key index path EQ, NOT_EQ, IN, NOT_IN, RANGE, IS_NULL, IS_NOT_NULL. Other predicate types fall through to expression scan. ## Test plan - [x] `MapFilterOperatorOpenStructTest` — 5 tests: - EQ on materialized key → PER_KEY_INDEX delegate - EQ on absent key (fully materialized) → EmptyFilterOperator (EOF) - IS_NULL on absent key (fully materialized) → MatchAllFilterOperator (numMatchingDocs == numDocs) - EQ on sparse key (not fully materialized) → EXPRESSION_FILTER fallback - IS_NOT_NULL with null bitmap → PER_KEY_INDEX, correct doc count (numDocs − nullCount) - [x] All existing pinot-core tests pass - [x] spotless/checkstyle clean -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
