peterxcli opened a new pull request, #5715: URL: https://github.com/apache/datafusion-comet/pull/5715
## Which issue does this PR close? Closes #5549. Third split of #5546, following merged #5552 (complete Arrow Field export) and #5631 (Variant type identity). Physical-reader compatibility follows in #5550; JVM scan admission and defaults follow in #5551. ## Rationale for this change A marked Variant Field can reach the native Parquet schema adapter as a Struct containing `metadata`, residual `value`, and optional `typed_value`. Spark's whole-value vector interface requires exactly two Binary children in `[value, metadata]` order. Reordering children alone cannot reconstruct shredded values. The adapter can also eliminate identity casts or leave identical physical and logical Fields as a bare column. A Variant with that layout still needs normalization, so the extension marker must select the conversion before those shortcuts. ## What changes are included in this PR? - Route explicitly marked Variant Fields through `CometCastColumnExpr`, including identical Fields and the adapter's complex-type fallback. - Isolate whole-value normalization in `cast_column/variant.rs`. Reuse the existing `parquet::variant` implementation to resolve storage children by name and call `unshred_variant` once. - Produce exactly `[value: Binary, metadata: Binary]`, remove `typed_value`, preserve parent nulls, and retain the original marked output Field. - Prepare legacy Spark UTF-16-ordered residual objects for Arrow's UTF-8 validation, including residuals inside shredded objects and lists. Convert completed values to the ordering required by supported Spark profiles; already compatible values retain their bytes. #5474 owns output-ordering cleanup after the Spark fix reaches every supported profile. - Reuse the shared Unicode Parquet field-name matching landed in #5602, which closed #5495. A regression verifies that logical `münchen` resolves to physical `MÜNCHEN` while keeping the normalization wrapper and output marker. This is native normalization infrastructure. JVM Variant scan admission remains closed, so this PR does not enable `SELECT v` on its own. Encoded children, physical-type coercions, empty-key metadata compatibility, reader-schema policy, and remaining field-ID handling belong to #5550. No dependency is added. ## How are these changes tested? Focused Rust tests cover canonical, fully shredded, and partially shredded values; out-of-order storage children; objects, arrays, scalars, JSON null, SQL null, nullable parents; nested residual metadata-row mapping; Unicode object ordering; idempotent normalization; and Unicode schema-name remapping. Validation: - `cargo test -p datafusion-comet variant --no-fail-fast`: 10 passed. - `cargo test -p datafusion-comet parquet::schema_adapter --no-fail-fast`: 30 passed. - `cargo clippy --color=never --all-targets --workspace -- -D warnings`: passed. - `cargo fmt --all -- --check` and `git diff --check upstream/main...HEAD`: passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
