peterxcli opened a new issue, #5978: URL: https://github.com/apache/datafusion-comet/issues/5978
### What is the problem the feature request solves? #5868 calls Arrow `unshred_variant`, then [`rebuild_spark_variant`](https://github.com/apache/datafusion-comet/blob/9a4cd83b59807522f4310778be78f9190a5dde59/native/core/src/parquet/cast_column/variant.rs#L981-L1020) to produce Spark-compatible bytes. The second traversal uses Arrow builders but reconstructs values and metadata, adding allocation and runtime cost. #5474 tracks UTF-16 object-key ordering. Resolving SPARK-58949 does not remove the other output requirements introduced by the review fixes. ### Describe the potential solution Produce Spark-compatible output in one reconstruction pass, preferably by reusing or extending the upstream builders/unshredder through an explicit compatibility policy. Preserve byte-level assertions. The replacement must preserve: - Spark's value-dependent integer and decimal widths, short-string encoding, and canonical typed NaN bits. - Metadata dictionary insertion/traversal order, field IDs, unused-key removal, and metadata flags. - Object ordering for every supported Spark version, including legacy-input handling. - Original scalar bytes for residual values and original value/metadata bytes for unshredded rows. - Nested objects/lists, empty keys, SQL NULL, Variant null, and malformed-input errors. Delete the extra reconstruction pass only after the native scan passes the existing strict Spark comparisons. Use the matched forward/reverse scan benchmark and native allocation benchmark to demonstrate the effect; keep benchmark results in the PR description. ### Additional context Parent: #5477. Related ordering-only cleanup: #5474. Current implementation: [`SparkOutputMetadata`, `spark_typed_scalar`, and `append_spark_variant`](https://github.com/apache/datafusion-comet/blob/9a4cd83b59807522f4310778be78f9190a5dde59/native/core/src/parquet/cast_column/variant.rs#L738-L979). Existing [scalar byte regression](https://github.com/apache/datafusion-comet/blob/9a4cd83b59807522f4310778be78f9190a5dde59/native/core/src/parquet/cast_column/variant/tests.rs#L235-L273) and `CometVariantProjectionSuite` cover the output contract. The measurements in [#5868](https://github.com/apache/datafusion-comet/pull/5868) establish a cost to address; they do not isolate the second traversal's share of that cost. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
