andygrove opened a new issue, #79: URL: https://github.com/apache/datafusion-spark/issues/79
### Describe the bug `format_string` raises an error for three kinds of input that Spark formats: 1. **`%b` / `%B` with an argument that isn't a boolean.** `ConversionType::validate` requires a `Boolean` argument ([format_string.rs#L584-L590](https://github.com/apache/datafusion-spark/blob/fc2babe8c153d95e70fd303d55368a800d90930c/crates/spark/src/function/string/format_string.rs#L584-L590)). Spark follows `java.util.Formatter`, which prints `false` for NULL and `true` for any other value. 2. **`%h` / `%H`.** These are parsed as `HexHashLower` and `HexHashUpper` ([format_string.rs#L791-L792](https://github.com/apache/datafusion-spark/blob/fc2babe8c153d95e70fd303d55368a800d90930c/crates/spark/src/function/string/format_string.rs#L791-L792)) but never formatted. Spark prints the hex hash code of the argument. 3. **A format that isn't a string.** `format_string(1)` is an error, while Spark casts the format to a string. ### To Reproduce Spark 4.2.0: ```sql SELECT format_string('Bool: %6b', 1); -- Bool: true SELECT format_string('Hash: %h', 'test'); -- Hash: ec06e15a SELECT format_string('Hash: %H', 'test'); -- Hash: EC06E15A SELECT format_string('Hash: %10h', 'test'); -- Hash: ec06e15a SELECT format_string(1); -- 1 ``` DataFusion, existing tests: each query expects an error, at [format_string.slt#L257-L258](https://github.com/apache/datafusion-spark/blob/fc2babe8c153d95e70fd303d55368a800d90930c/crates/sqllogictest/test_files/string/format_string.slt#L257-L258), [#L1082-L1083](https://github.com/apache/datafusion-spark/blob/fc2babe8c153d95e70fd303d55368a800d90930c/crates/sqllogictest/test_files/string/format_string.slt#L1082-L1083) and [#L1809-L1822](https://github.com/apache/datafusion-spark/blob/fc2babe8c153d95e70fd303d55368a800d90930c/crates/sqllogictest/test_files/string/format_string.slt#L1809-L1822). The `%h` test already has Spark's result in a comment. ### Expected behavior Format these the way Spark does, and change the expectations listed above. For `%h`, Spark hashes its internal value, so matching it means reproducing Spark's `hashCode`, for example `UTF8String.hashCode` for strings. If that is too much, `%h` could stay an error, with the difference documented. ### Additional context Found by running the sqllogictest files on Spark 4.2.0 with the PySpark runner from #54 (ANSI mode off, UTC session). The Spark results above come from that run. DataFusion results are the expectations in the `.slt` files at fc2babe8, which CI enforces. #14 covers the `%t` specifiers, and #39 covers `%s` of a DECIMAL and `%g`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
