comphead opened a new issue, #26340: URL: https://github.com/apache/datafusion/issues/26340
### Describe the bug Spark's `UnBase64` decodes with `java.util.Base64.getMimeDecoder` ([stringExpressions.scala#L2897-L2918](https://github.com/apache/spark/blob/v4.0.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala#L2897-L2918)). That decoder does not require padding and ignores characters outside the Base64 alphabet. `unbase64` in `datafusion-spark` is rewritten to the core `decode(..., 'base64pad')` ([base64.rs#L169-L172](https://github.com/apache/datafusion/blob/9dec77584ea39d93aa01bba7f880b318b37d2c24/datafusion/spark/src/function/string/base64.rs#L169-L172)), and the existing test expects an error for input that Spark decodes: [base64.slt#L109](https://github.com/apache/datafusion/blob/9dec77584ea39d93aa01bba7f880b318b37d2c24/datafusion/sqllogictest/test_files/spark/string/base64.slt#L109) has `unbase64('123')` failing with "Failed to decode value using base64", and [#L112](https://github.com/apache/datafusion/blob/9dec77584ea39d93aa01bba7f880b318b 37d2c24/datafusion/sqllogictest/test_files/spark/string/base64.slt#L112) does the same for the binary variant. ### To Reproduce Spark 4.0.0, the same result with `spark.sql.ansi.enabled` on or off for `'123'` and `'YWJj!!'`: ```sql SELECT hex(unbase64('123')) -- D76D SELECT hex(unbase64('12')) -- D7 SELECT hex(unbase64('YQ')) -- 61 SELECT hex(unbase64('YW Jj')) -- 616263 SELECT hex(unbase64('YWJj!!')) -- 616263 SELECT hex(unbase64('!!!!')) -- (empty) SELECT hex(unbase64('Y')) -- error: IllegalArgumentException: Last unit does not have enough valid bits ``` Only `unbase64('123')` was run on DataFusion, through the existing test. The other inputs are listed for Spark's behavior and were not run. ### Expected behavior Decode like the MIME decoder: accept missing padding, skip characters outside the alphabet, and keep an error for a dangling single character. Update the expectations at `base64.slt#L109` and `#L112`. ### Additional context Found by running the queries in `datafusion/sqllogictest/test_files/spark` on a local Spark and comparing the results with the expected values in the files, which is the kind of check proposed in #17045. Spark results come from Apple's build of Spark 4.0.0 (`4.0.0.65-apple`) with a UTC session and `spark.sql.ansi.enabled=false` unless stated. The Spark source links point at the `v4.0.0` tag. DataFusion results are the sqllogictest expectations at 9dec77584, which CI enforces. Anything marked as not run was derived from reading the code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
