andygrove commented on issue #26337:
URL: https://github.com/apache/datafusion/issues/26337#issuecomment-6698731421
Running `format_string.slt` on Spark 4.2.0 with the runner in #27165 found
these queries, which raise an error in DataFusion and return a value in Spark:
```sql
SELECT format_string('Bool: %6b', 1) -- Bool: true
SELECT format_string(1) -- 1
SELECT format_string('Hash: %h', 'test') -- Hash: ec06e15a
SELECT format_string('Hash: %H', 'test') -- Hash: EC06E15A
SELECT format_string('Hash: %10h', 'test') -- Hash: ec06e15a
```
Java's `%b` prints `true` for any non-null argument that is not a boolean,
and `%h` prints the argument's hash code, here the hash of Spark's internal
`UTF8String`.
It also happens the other way round. Spark passes its internal values to
`java.util.Formatter`
([stringExpressions.scala#L2086](https://github.com/apache/spark/blob/v4.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala#L2086)),
so `%f`, `%e`, `%g` and `%a` fail for a `DECIMAL` argument with `f !=
org.apache.spark.sql.types.Decimal`. The tests pass literals such as `1.5`,
which are `DECIMAL` in Spark and `DOUBLE` in DataFusion (#27185), so 61 of them
raise this error in Spark.
Every Spark version from 3.4 to 4.2 passes internal values to
`java.util.Formatter` the same way, so none of this, nor the `%t` behavior in
#21515, depends on the Spark version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
