peterxcli opened a new issue, #5546: URL: https://github.com/apache/datafusion-comet/issues/5546
### What problem does this solve? #5407 implements the ordinary-Parquet phase of #4295, but it now combines several independently reviewable concerns: shared Arrow FFI schema export, Variant logical-type transport, native whole-value normalization, Spark-compatible Parquet physical semantics, and scan admission/schema-evolution defaults. This issue tracks re-landing that work as five atomic pull requests under the broader Variant roadmap in #5438, following the maintainer request in https://github.com/apache/datafusion-comet/pull/5407#issuecomment-5463550828. ### Scope Support direct, top-level, whole-value `VariantType` projection from ordinary Parquet: ```sql SELECT v FROM parquet_table; SELECT id, v, tail FROM parquet_table; ``` The completed series preserves Variant identity through: ```text Spark VariantType -> Comet protobuf -> marked Arrow Field -> ordinary native Parquet scan -> whole-value normalization -> Arrow C Data Interface -> CometStructVector -> ColumnVector.getVariant ``` This tracker does not include Iceberg, nested Variant, Variant expressions, subfield pruning, predicate pushdown, writes, columnar-to-row, shuffle/spill, or Python transport. ### Ordered subtasks - [ ] Export complete Arrow Fields through the native FFI boundary - [ ] Carry Spark VariantType identity through Comet schema serialization - [ ] Normalize marked Variant arrays at the native Parquet boundary - [ ] Match Spark physical Parquet semantics for projected Variant columns - [ ] Enable direct top-level Variant scan admission, defaults, and fallbacks Each pull request must include focused tests and retain explicit Spark fallback until the complete scan boundary is ready. ### Definition of done - Direct top-level Variant projection remains a Comet native Parquet scan and matches Spark. - Fields before and after Variant retain their order. - Objects, arrays, scalars, Variant JSON null, SQL NULL, and nullable parents round-trip. - Canonical unshredded and shredded inputs reconstruct whole values. - Spark receives logical `VariantType` with exactly `[value, metadata]` children. - Unread Variant pruning from #5377 remains native. - Unsupported expressions, nested Variant, pushed `VariantStruct`, native writes, C2R, shuffle/spill, Python operators, and Iceberg retain explicit fallback. - Spark 3.x compilation and behavior remain unchanged. ### Related work - #3983 — shredded writing, pruning, and predicate pushdown - #5474 — removal of legacy Spark UTF-16 output rewriting - #5477 — removal or reassessment of upstream compatibility paths - #5495 — Spark-compatible Unicode Parquet field matching - #5519 — pushed one-field `VariantStruct` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
