andygrove opened a new issue, #91: URL: https://github.com/apache/datafusion-spark/issues/91
`elt` always returns `Utf8`: `return_type` returns `Utf8`, and `coerce_types` casts every value argument to `Utf8` (`crates/spark/src/function/string/elt.rs`). Spark returns `BINARY` when every value argument is binary. In Spark, [`EltTypeCoercion`](https://github.com/apache/spark/blob/v4.1.1/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionHelper.scala#L476-L490) casts the value arguments to `STRING` only if `spark.sql.function.eltOutputAsString` is true or at least one of them is not binary. [`Elt.dataType`](https://github.com/apache/spark/blob/v4.1.1/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala#L299-L300) is the type of the first value argument. `spark.sql.function.eltOutputAsString` defaults to false. ```sql SELECT arrow_typeof(elt(1, X'61', X'62')); -- datafusion-spark: Utf8 -- Spark: BINARY ``` I checked the datafusion-spark result through the sqllogictest runner, and the Spark behavior in source; I did not run it on PySpark. `concat` already handles the same case correctly: it returns `Binary` for all-binary input, which matches Spark's default for `spark.sql.function.concatBinaryAsString`. ### Fix 1. Keep binary when every value argument is binary, in both `coerce_types` and the return type. 2. Optionally support `spark.sql.function.eltOutputAsString = true`. That flag changes the return type, so `elt` needs it when it is constructed, set from the config in `with_updated_config`. This needs `SparkOptions` to move into this crate. Part of the Spark-version and legacy-setting work in #16. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
