raghuvanshraj opened a new issue, #26517: URL: https://github.com/apache/datafusion/issues/26517
### Is your feature request related to a problem or challenge? ## Is your feature request related to a problem or challenge? OpenSearch is adding a Variant field type to its Parquet-backed indices. The query engine is DataFusion, the writer and Variant kernels are arrow-rs, and the SQL functions are `datafusion-contrib/datafusion-variant`. Files mix shredded and unshredded layouts within one table, and the same table is read by Spark. This epic lists the core hooks that work still needs, following the split in #16116: generic extension points in core, Variant-specific implementations in the crate. It sits under #16116 and #21308. Items already tracked elsewhere are linked rather than restated; the new issues are the ones marked **new**. Spelling the type in SQL (section 6) already has upstream issues filed; they are linked here so the epic covers the whole core surface Variant needs. ## Describe the solution you'd like ### 1. Read only the leaves a UDF needs `variant_get(v, '$.x')` reads and decodes the whole Variant group because the Parquet planner only trusts `get_field`. Two PRs address this: #25013 (two hooks on `ScalarUDFImpl`) and draft #26282 (`NestedInputAccess` capability, also used by `get_field`, with row group and page pruning on the declared leaves). Our requirement for whichever shape lands: a function must be able to require more than one leaf for one argument (the typed leaf and the `value` fallback leaf), and pruning must keep any container that holds a fallback row. #26282 does both. - [ ] Review #25013 / #26282 against shredded and unshredded test data and help converge on one shape - [ ] Row group and page pruning on a declared leaf, with the fallback guard: #20871. #26282 includes it; if the work is split, this is the successor to #25013 - [ ] `datafusion-variant`: implement the capability for `variant_get` once it lands ### 2. Keep the extension tag through planning A Variant is a struct whose field metadata carries `ARROW:extension:name = arrow.parquet.variant`. Two core paths drop it, after which `variant_get` sees a plain struct and rejects the column. - [ ] **new** Parquet schema inference clears all field metadata before `Schema::try_merge` when `skip_metadata = true` (the default), so any `ListingTable` over more than one file loses the tag (`datasource-parquet/src/file_format.rs`, `clear_metadata`). Reproduced on 54. Proposed fix: keep the `ARROW:extension:*` keys recursively and drop the rest; the extension keys are the type's identity, not writer annotation. - [ ] `CASE` / `COALESCE` build the output `Field` from the `DataType` alone: #24860 (reported on `geoarrow.wkb`, same mechanism) ### 3. Tables that mix shredded and unshredded files When one file stores `v` as `Struct<metadata, value>` and another as `Struct<metadata, value, typed_value: Struct<x: Int64>>`, the merged table schema has `typed_value`, and the default adapter null-fills it for the unshredded file (`physical-expr-adapter/src/schema_rewriter.rs`). For an ordinary struct that is correct. For Variant the shredding spec makes `value = object, typed_value = null` an invalid row, so a compliant reader returns NULL for every field of every row of that file, silently. Reproduced: a two-file table returned 0 of 1024 unshredded rows. Files shredded on different keys fail differently, at open, with `target field is non-nullable but missing from source`. The constraint we would like core to hold: a file is read in its own layout and reconstructed, never null-filled into a layout it does not have. Two ways to satisfy it: per-file `required_fields` answers (#26282 already lets the answer differ per file; to be confirmed whether that alone avoids the null-fill), or a hook for a registered extension type to supply its own per-file adaptation. The Variant adapter itself would live in the crate. Named earlier on #16116 (2025-07-15) and #22908 (2026-09-08). - [ ] **new** Per-file adaptation for extension types, so an unshredded file is never null-filled into the shredded layout: per-file `required_fields` answers (#26282) or an extension-type adapter hook, with the Variant adapter in the crate ### 4. Statistics the optimizer can use `ColumnStatistics` is flat, so a nested leaf (shredded or not) and any UDF output have no planner statistics. Two designs are open for the UDF half and one has to be picked; #26282 adds a container-level `output_statistics` for pruning, which is a third data point. - [ ] Nested and extension-type leaves in `Statistics`: #20707 - [ ] UDF output statistics: `ScalarUDFImpl::output_statistics` (#21307) or `ExpressionAnalyzer` (#21122). We implement whichever is chosen; the crate half (typed-leaf min/max, fallback-leaf null count) is the same code either way. ### 5. Variant value semantics `GROUP BY`, `DISTINCT`, joins and `ORDER BY` on a Variant column are undefined today. Spark rejects all of them at analysis time. DuckDB 1.5 allows them, but `=`, `GROUP BY`, `DISTINCT` and `ORDER BY` each apply a different rule (equal numbers of different width compare equal but group apart; `ORDER BY` is by inner type tag, so `7.0` sorts after `42`). - [ ] **new, after design discussion** A canonical-bytes normalisation kernel in arrow-rs so equal values have equal bytes, and a `variant_normalize` function in the crate on top of it, so hashing operators work without each inventing a rule. For `ORDER BY` our lean is to refuse at plan time as Spark does. Issue to follow once the SQL semantics are written up. ### 6. Spelling `VARIANT` in SQL The generic machinery is merged: `ExtensionTypeRegistry` and `DFExtensionType` (#20312, DataFusion 54), `TypePlanner` (#20676, 54), and metadata-preserving `CAST` (#23169, 55.1.0 via backport #24875). What is missing is the Variant entries. Without them a Variant prints as raw struct bytes in `datafusion-cli`, and `CREATE TABLE t (v VARIANT)` or `CAST(x AS VARIANT)` cannot be planned. Spark 4.2 and DuckDB 1.5 both spell the type `VARIANT` in `CREATE TABLE` and `CAST`, so this is what lets a query written for them run unchanged. - [ ] Register `arrow.parquet.variant` as a `DFExtensionType` (JSON display, and the identity the other hooks key on): the one unticked box in #21144. Small; the natural first PR - [ ] Map the SQL name `VARIANT` to that type via `TypePlanner`: #21626, same shape as UUID and JSON. The crate-side registration goes with it - [ ] Pluggable cast resolution to and from extension types (`CAST(x AS VARIANT)`, `v::INT`): #20755, after the registration lands and a maintainer design. Not on our critical path ### Adjacent, in arrow-rs - [ ] The Parquet writer omits `specification_version` from the `VARIANT` logical type (`LogicalType::variant(None)`), so Spark 4.1 rejects the file with `PARQUET_TYPE_NOT_RECOGNIZED`. Reproduced 2026-09-25; unchanged on arrow-rs main as of 2026-10-19 (`parquet/src/arrow/schema/extension.rs`, `logical_type_for_struct`). One-line fix; issue to be filed on arrow-rs and linked here. ## Describe alternatives you've considered Keeping all of this inside OpenSearch (a fork of the Parquet reader and a private `variant_get`). Rejected: every hook above is generic to extension types, and the mixed-layout and statistics problems are hit by any Variant reader on DataFusion, not just ours. ## Additional context Compatibility target for the crate work is Spark 4.2.0 (datafusion-variant#69). We build against DataFusion 54 today and are moving to 55 / arrow-rs 59, which the crate requires. Open questions for maintainers: 1. Are items 2 and 3 accepted as core concerns (neither mentions Variant), with the Variant adapter shipped from the crate? 2. For item 1, which shape: #25013, #26282, or a merge of the two? 3. For item 4, #21307 or #21122? ### Describe the solution you'd like _No response_ ### Describe alternatives you've considered _No response_ ### Additional context _No response_ ### Willingness to contribute None -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
