tarun11Mavani opened a new pull request, #18710: URL: https://github.com/apache/pinot/pull/18710
OPEN_STRUCT infrastructure — SPI utilities, index registration, stats constructors (PR 2/4) --- ## Summary This is **PR 2a** in the OPEN_STRUCT stack — the infrastructure/setup slice split from #18643 (storage layer) to make review more manageable. It adds foundational pieces that the ingestion/load pipeline (PR 2b) builds on. RFC: https://docs.google.com/document/d/14kPmjDTKbO8l0ql4rrN7I5Yki5pqMw6GeGmxxc9grsU/edit?tab=t.0 PR 1 (SPI + data model): #18368 — merged ### What's in this PR **SPI utilities (pinot-spi):** - `OpenStructTypeInference` — value→DataType inference for OPEN_STRUCT keys (TIMESTAMP retained; DATE/TIME/UUID → STRING; Byte/Short → INT; unrepresentable → null) - `OpenStructNaming` — `isMaterializedOpenStructColumn()`, `parseParentColumn()`, `parseKey()` name-based child column detection helpers - `PinotDataType` — OPEN_STRUCT→MAP ingestion coercion in `getPinotDataTypeForIngestion()` **SPI utilities (pinot-segment-spi):** - `FieldIndexConfigsUtil.fromFieldConfig()` — builds `FieldIndexConfigs` from a single `FieldConfig` without requiring a `TableConfig`/`Schema`, enabling per-key index resolution for synthetic materialized children - `V1Constants` — `PARENT_COLUMN`, `HAS_SPARSE_COLUMN` metadata keys - `OpenStructDataSource` — default `getMapValue(docId)` SPI method **Index registration (pinot-segment-local):** - `OpenStructIndexType` + `OpenStructIndexPlugin` — `@AutoService(IndexPlugin.class)` registration, per-key index validation against vetted allowlist, no-op reader factory, no-op index handler - `OpenStructSupportedIndexes` — vetted allowlist of per-key index types (forward, dictionary, inverted, range, bloom) **Stats collector infrastructure:** - `AbstractColumnStatisticsCollector` — new `(FieldSpec, FieldConfig, PartitionFunction)` constructor for creating collectors without `StatsCollectorConfig`/`Schema` - 7 scalar collectors (`Int`, `Long`, `Float`, `Double`, `BigDecimal`, `String`, `Bytes`) — delegating FieldSpec constructors - `StatsCollectorUtil` — OPEN_STRUCT cases + `createStatsCollector(FieldSpec, FieldConfig)` overload **Wiring hooks:** - `ForwardIndexType` — `shouldCreateIndex()` returns false for OPEN_STRUCT parent columns - `TableConfigUtils` — rejects user columns containing the reserved `$` separator when any field is OPEN_STRUCT ### What's NOT in this PR (deferred to PR 2b) - Mutable path: `MutableOpenStructIndex`, `MutableKeyColumn`, `MutableOpenStructDataSource` - Seal path: `OpenStructColumnSplitter` (the core dense/sparse classification and materialization) - Immutable path: `ImmutableOpenStructDataSource`, segment load wiring - All wiring into `MutableSegmentImpl`, `ImmutableSegmentImpl`, `BaseSegmentCreator`, `SegmentColumnarIndexCreator`, etc. - Integration tests ### Backward compatibility - Zero behavior change for existing schemas and segments - OPEN_STRUCT is opt-in at schema authoring time - The `$` column name restriction applies only to schemas containing at least one OPEN_STRUCT field ## Test plan - [x] `OpenStructTypeInferenceTest` — value→DataType mapping for all input types including widening, folding, and unrepresentable - [x] `OpenStructNamingTest` — materialized/sparse column naming, `isMaterializedOpenStructColumn`, `parseParentColumn` parsing - [x] `FieldIndexConfigsUtilTest` — `fromFieldConfig` with dictionary/raw encoding, inverted/range/bloom indexes, default fallback - [x] `StatsCollectorUtilFieldSpecTest` — FieldSpec-based collector factory for all scalar types - [x] `OpenStructIndexTypeTest` — index registration, no-op handler, config extraction, per-key index validation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
