peterxcli opened a new issue, #5977: URL: https://github.com/apache/datafusion-comet/issues/5977
### What is the problem the feature request solves? #5868 adds checks in [`rewrite_shredding_state`](https://github.com/apache/datafusion-comet/blob/9a4cd83b59807522f4310778be78f9190a5dde59/native/core/src/parquet/cast_column/variant.rs#L226-L242) for missing required values and invalid null shredding-state structs. Comet reports `MALFORMED_VARIANT`, while missing object fields remain valid and parent SQL NULL masks its children. This behavior follows Spark's reader. The [Parquet specification](https://github.com/apache/parquet-format/blob/master/VariantShredding.md#value-shredding) requires readers to return Variant null for some missing required-value states. An ordinary Arrow upgrade therefore does not establish that Comet's checks can be deleted. ### Describe the potential solution Define and test the Spark compatibility policy at a shared native validation boundary, using upstream support where appropriate. Preserve the canonical upstream policy when it differs from Spark. Completion requires: - Distinguish an absent object field, a missing required top-level value, an invalid list element/state struct, SQL NULL, and explicit Variant null. - Match the result or error from each supported Spark version, including parent-null masking. - Remove duplicated Comet prevalidation only when the replacement enforces those same rules. - Keep conversion to Spark's `MALFORMED_VARIANT` error in the Comet integration layer. - Verify behavior through native Parquet scans and the unchanged upstream Spark assertions. ### Additional context Parent: #5477; implementation: #5868. Existing regression: [`normalize_rejects_missing_required_shredding_states`](https://github.com/apache/datafusion-comet/blob/9a4cd83b59807522f4310778be78f9190a5dde59/native/core/src/parquet/cast_column/variant/tests.rs#L434-L472). apache/arrow-rs#10619 addresses a different malformed object-field layout and does not resolve this policy difference. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
