peterxcli opened a new pull request, #5868: URL: https://github.com/apache/datafusion-comet/pull/5868
## Which issue does this PR close? Closes #5551. ## Rationale for this change Complete the ordinary-Parquet admission step in #5546 so Spark can project whole Variant values through a native scan. ## What changes are included in this PR? Enable direct top-level Variant projection with `allowReadingShredded=true` and `pushVariantIntoScan=false`. Carry missing-column defaults as constant `[value, metadata]` structs while preserving their schema indexes, and retain Spark fallbacks for unsupported consumers and reader settings. Reuse the Variant normalizer to handle Spark's empty object keys until Comet upgrades to an Arrow release containing apache/arrow-rs#10352. Document the supported surface and add removal TODOs for the compatibility paths, including #5477. ## How are these changes tested? Focused Spark 4.0/4.1 tests cover value parity, defaults, Unicode field matching, vector layout, and fallback behavior. Native Variant tests and clippy pass, and Spark 3.5 compiles. The new projection suite runs in Linux and macOS CI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
