andygrove opened a new issue, #27186: URL: https://github.com/apache/datafusion/issues/27186
### Describe the bug `encode` follows Spark 3.5 ([encode.rs#L22-L43](https://github.com/apache/datafusion/blob/a616831122aa525ed843ee3c5e11f4803f557c98/datafusion/spark/src/function/string/encode.rs#L22-L43)). Spark 4.0 changed it in two ways, which the code lists as not implemented: 1. Only `us-ascii`, `iso-8859-1`, `utf-8`, `utf-16be`, `utf-16le`, `utf-16` and `utf-32` are accepted ([CharsetProvider.scala#L28-L29](https://github.com/apache/spark/blob/v4.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/CharsetProvider.scala#L28-L29)). Any other name, such as `utf-32be`, raises `INVALID_PARAMETER_VALUE.CHARSET`. 2. A character that the charset cannot represent raises `MALFORMED_CHARACTER_CODING` instead of becoming `?`. ### To Reproduce Spark 4.2.0: ```sql SELECT encode('A', 'utf-32be') -- [INVALID_PARAMETER_VALUE.CHARSET] The value of parameter(s) `charset` in `encode` is invalid: expects one of the iso-8859-1, us-ascii, utf-16, utf-16be, ... SELECT encode('é', 'us-ascii') -- [MALFORMED_CHARACTER_CODING] Invalid value found when performing `encode` with us-ascii SQLSTATE: 22000 ``` DataFusion returns `00000041` and `3f` ([encode.slt#L42](https://github.com/apache/datafusion/blob/a616831122aa525ed843ee3c5e11f4803f557c98/datafusion/sqllogictest/test_files/spark/string/encode.slt#L42-L45), [#L66](https://github.com/apache/datafusion/blob/a616831122aa525ed843ee3c5e11f4803f557c98/datafusion/sqllogictest/test_files/spark/string/encode.slt#L66-L69)). ### Spark versions The result depends on the Spark version: | Spark | Charsets accepted | Unmappable character | | --- | --- | --- | | 3.4, 3.5 | any Java charset name, through `String.getBytes` | replaced with `?` | | 4.0, 4.1, 4.2 | the seven above | `MALFORMED_CHARACTER_CODING` | In 4.x, `spark.sql.legacy.javaCharsets` and `spark.sql.legacy.codingErrorAction` restore the 3.5 behavior. Both default to false. ### Expected behavior `datafusion-spark` has no notion of a Spark version (#21698), so `encode` should either document that it is compatible with Spark 3.5, or have settings that select the behavior, such as `datafusion.spark.*` options that mirror the two legacy settings, so that it can support both 3.x and 4.x. The TODO in `encode.rs` suggests the second. ### Additional context Found by running the Spark `.slt` files on PySpark 4.2.0 with the runner in #27165, which reports the same for `utf-32le` and `iso-8859-1`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
