peterxcli opened a new issue, #5437: URL: https://github.com/apache/datafusion-comet/issues/5437
### What is the problem the feature request solves? Spark 4.1 supports top-level Variant values through Arrow Python transport. Its JVM Arrow bridge maps Variant to a `Struct<value, metadata>` and marks the metadata child ([ArrowUtils](https://github.com/apache/spark/blob/v4.1.3/sql/api/src/main/scala/org/apache/spark/sql/util/ArrowUtils.scala#L183-L197)); PySpark converts that shape to and from `VariantVal` ([PySpark Arrow conversion](https://github.com/apache/spark/blob/v4.1.3/python/pyspark/sql/pandas/types.py#L201-L208), [Variant value conversion](https://github.com/apache/spark/blob/v4.1.3/python/pyspark/sql/pandas/types.py#L1157-L1171)). Comet currently excludes any Variant-bearing input or output from its accelerated MapInArrow/MapInPandas rewrite in [`EliminateRedundantTransitions`](https://github.com/peterxcli/datafusion-comet/blob/c355fefd9c0b7e96d86523a7214bb2cdd47e1a55/spark/src/main/scala/org/apache/comet/rules/EliminateRedundantTransitions.scala#L203-L213). This is a safe fallback, but it leaves the Python batch transport capability untracked. ### Describe the potential solution - Support direct, top-level Variant inputs and outputs in Comet MapInArrow and MapInPandas operators. - Translate only at the Python boundary between Comet's canonical parent `arrow.parquet.variant` marker and Spark/PyArrow's Variant child marker. - Preserve exact `[value, metadata]` order, Binary children, field names, SQL-null parent bitmap, Variant JSON null, and zero-row streams in both directions. - Reject malformed layouts and retain fallback for nested Variant or unsupported Python operator families. Add parity and plan tests for MapInArrow and MapInPandas identity functions, projected fields around Variant, nullable values, and Python-produced Variant output. Verify Spark receives `VariantType`, not an ordinary Struct. ### Additional context #5386 tracks scalar Arrow Python UDF expansion and #5123 tracks grouped/window/apply operators generally; neither specifies Variant's logical metadata and layout. #5407 and #5425 define the canonical Comet-side Field contract. Nested Variant and the other Python operator families remain separate until explicitly supported. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
