peterxcli opened a new issue, #5430: URL: https://github.com/apache/datafusion-comet/issues/5430
### What is the problem the feature request solves? Spark 4.x supports `CAST` / `TRY_CAST` between `VariantType` and a defined set of scalar and nested types, but Comet explicitly rejects every pair involving Variant in [`CometCast`](https://github.com/peterxcli/datafusion-comet/blob/c355fefd9c0b7e96d86523a7214bb2cdd47e1a55/spark/src/main/scala/org/apache/comet/expressions/CometCast.scala#L175-L187). Examples that therefore fall back include: ```sql SELECT CAST(v AS STRING), TRY_CAST(v AS ARRAY<INT>) FROM t; SELECT CAST(id AS VARIANT), CAST(array(1, 2) AS VARIANT) FROM t; ``` Spark admits casts from Variant through [`VariantGet.checkDataType`](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L127-L142) and supports the reciprocal cast to Variant for scalar values and supported arrays. Structs and maps use `to_variant_object` instead. The runtime routes Variant casts through Spark's [Variant cast implementation](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L1239-L1258), including [generated execution](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L1343-L1366). ### Describe the potential solution Add only Spark-supported Variant pairs to Comet's cast matrix and native cast implementation: - reuse the extraction/conversion semantics from #5424 for casts from Variant, including nested targets once supported; - encode casts to Variant with canonical `[value, metadata]` storage and the Variant result Field contract from #5425; - preserve SQL NULL versus Variant JSON null, decimal, binary, date/timestamp, time-zone, overflow, and recursive null behavior; - match `CAST` versus `TRY_CAST` error behavior; and - keep casts that Spark does not admit unsupported. In particular, maps and structs must use `to_variant_object`, not a broader Variant cast. Add focused parity and native-plan tests across supported scalar types, arrays and nested targets, SQL/JSON nulls, invalid conversions, `TRY_CAST`, aliases, and columns around a Variant-producing result. Spark 3.x compilation and behavior must remain unchanged through the version shim. ### Additional context Related work: #4295, #5407, #5424, #5425, and the dedicated `to_variant_object` tracker. Non-goals: comparisons and hash expressions (Spark rejects Variant for both), `to_variant_object`, parsing JSON, shuffle/spill, C2R, writing, and Python transport. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
