xiangfu0 opened a new pull request, #19101: URL: https://github.com/apache/pinot/pull/19101
## Summary Add an end-to-end `VARIANT` user journey for Apache Pinot: - define a single-value `VARIANT` dimension that is retained in Pinot's raw forward index - ingest top-level, non-repeated Apache Parquet `VARIANT(1)` values, including unshredded and shredded layouts - materialize frequently queried paths during ingestion while retaining the original Variant value - expose Spark-style parse, extraction, type, null, and JSON-rendering functions in both query engines - carry Variant through Pinot's schema, data block, DDL, JDBC, and response layers - add a packaged quickstart and an integration suite covering table creation, Parquet ingestion, and queries Raw Variant values are deliberately not orderable, comparable, or generally aggregatable. Pinot allows `COUNT(raw_variant)` but rejects raw comparison, grouping, distinct, ordering, join-key, set-operation, and other aggregate uses with guidance to extract a typed scalar first. The persisted format, null semantics, ownership boundaries, compatibility contract, and activation/rollback gates are recorded in [`pinot-spi/VARIANT_DESIGN.md`](https://github.com/xiangfu0/pinot/blob/xiangfu0/variant-e2e/pinot-spi/VARIANT_DESIGN.md). ## Try it Build Pinot and launch the dedicated batch quickstart: ```shell ./mvnw clean install -DskipTests -Pbin-dist -Pbuild-shaded-jar build/bin/quick-start-variant-batch.sh ``` The quickstart: 1. registers the `variantEvents` schema and offline table 2. ingests the committed Parquet `VARIANT(1)` fixture 3. uploads a five-row segment 4. runs the representative queries below ```sql SET enableNullHandling=true; SELECT eventType, COUNT(*) FROM variantEvents GROUP BY eventType ORDER BY eventType; SELECT eventId, variant_get(payload, '$.user.id', 'STRING') AS userId, variant_get(payload, '$.amount', 'DOUBLE') AS amount FROM variantEvents WHERE eventType = 'checkout' ORDER BY eventId; SELECT eventId, variantToJson(payload) FROM variantEvents ORDER BY eventId; ``` The complete sample is under `pinot-tools/src/main/resources/examples/batch/variantEvents`. ## Supported scope - top-level, non-repeated Parquet `VARIANT(1)` columns - a single-value Pinot `VARIANT` dimension stored without a dictionary - raw forward-index retention plus optional ingestion-time scalar materialization - storage null handling enabled by the schema or table - query-time typed extraction and JSON rendering with `SET enableNullHandling=true` - both single-stage and multi-stage query execution Nested/repeated Variant columns, streaming ingestion, quoted path keys, and mixed-version queries over active Variant tables are outside this initial verified scope. The automatic Parquet reader selection keeps the existing Avro-metadata precedence. Files that contain Avro metadata must explicitly select the native Parquet reader to ingest a Variant column. ## Production safety - centralizes Pinot's `PVAR` framing and validation in `VariantEnvelope`, freezes the version-1 bytes with a golden test, and assigns a new protobuf wire value without renumbering existing types - reports an actionable full-upgrade error when an unknown query-wire type is received - rejects schemas/tables without effective storage null handling and rejects VARIANT SQL functions when query null handling is disabled - rejects unsupported index configurations and unsafe raw Variant query operations, including window partition/order keys and non-`COUNT` raw aggregates - preserves historical Parquet reader selection, uses locale-independent type canonicalization, and converts Parquet dates to UTC epoch-day timestamps - initializes Parquet converters atomically, publishes replacement readers transactionally, caches immutable Variant schema indexes, preserves progress when a malformed row is skipped, and tests terminal malformed-row behavior - validates Parquet decimal precision, scale, byte bounds, and hostile exponents before allocation - writes array-backed, direct, and read-only Parquet `Binary` values into the final `PVAR` envelope without a full-payload intermediate copy and without changing source buffer positions or limits - compiles query paths and target types once, reuses allocation-free cursors, vectorizes existence checks, and shares cache/null lifecycle across the single-stage Variant functions - parses literal `parseJson`/`tryParseJson` inputs once per query in both engines and specializes ingestion-time Variant functions with constant paths/types into compiled, cursor-reusing evaluators - keeps tolerant extraction mismatch and numeric overflow on a non-throwing path, while preserving strict conversion behavior, and avoids `BigInteger` allocation for JSON integers that fit primitive ranges - restores the Java client's legacy textual JSON-null contract while distinguishing SQL null, encoded Variant null, and the Variant string `"null"` across Arrow, JSON, HTTP JDBC, and gRPC JDBC results - keeps only `parquet-variant` in `pinot-common`; Parquet column/schema dependencies remain isolated to the input plugin - adds approximately 124 KiB to the shaded common artifact, with no Parquet column/schema classes - covers raw forward-index min/max behavior and direct-buffer inputs with regressions - documents the activation contract: upgrade the complete Pinot fleet and external ingestion jobs before registering a Variant schema, and do not roll back while a Variant table is active `japicmp` reports no binary-incompatible changes for `pinot-spi` or `pinot-segment-spi`. The published Java 11 artifacts retain classfile major version 55. ### Compatibility and rollout Existing schemas, segments, and data types keep their current behavior. Older components reject an unknown Variant wire type deterministically, but they cannot operate on an active Variant table. Upgrade controllers, brokers, servers, clients, and external ingestion jobs/plugins before creating a Variant schema; do not activate Variant during a rolling mixed-version window. ## Verification - focused wire, utility, transform, validation, planner, runtime, client, DDL, and Parquet reader suites - 19 `VariantTypeTest` integration scenarios across both query engines - 63-module integration-test reactor build - 83-module clean shaded binary distribution build on JDK 25 (`BUILD SUCCESS`, 11m24s) - Java 11 compatibility scan: classfile major version 55 for SPI, segment SPI, common, time-series SPI, Java client, and JDBC client - packaged quickstart: table creation, five-row Parquet ingestion, materialized-field grouping, typed nested extraction, JSON rendering, SQL/Variant null semantics, and clean shutdown - local JMH smoke: compiled reusable extraction is 90.953 ns/op with 0.001 B/op versus 92.328 ns/op and 112.001 B/op for the object-returning path - local JMH smoke: unshredded Parquet conversion is 27.682, 29.470, and 30.850 ns/op for array-backed, direct, and read-only buffers respectively; all allocate the same 328 B/op final result - spotless, checkstyle, license formatting, and license checks across all 16 affected modules - `git diff --check` ## Review notes This is a design and implementation draft, not a merge-ready ownership approval. It changes public SPI (`FieldSpec.DataType.VARIANT`, `VariantEnvelope`, protobuf/DataSchema mappings) and requires explicit compatibility sign-off before merge: - [ ] SPI, schema, and query-wire owners - [ ] Parquet and input-format plugin owners - [ ] single-stage and multi-stage query-engine owners - [ ] Java, JDBC, Arrow, JSON, and gRPC client/response owners The draft is kept end-to-end so those owners can review one complete activation contract: the public type, its only supported ingestion path, the raw-value safety guards, and the executable acceptance test. No partial subset is safe to activate: the public type without ingestion is unusable, ingestion without all query guards exposes byte-layout semantics, and any subset without mixed-version gates is unsafe to roll out. The design document records the dependency order and the rule that VARIANT remains unactivatable until all safety pieces land. If maintainers prefer smaller rollback units after design review, this draft can be converted into that documented dependency-ordered stack: foundation and wire contract; segment and Parquet ingestion; query functions and engines; clients and DDL; then quickstart, integration, and benchmarks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
