andygrove commented on issue #95: URL: https://github.com/apache/datafusion-spark/issues/95#issuecomment-6855889831
### Literal types 42 of the 119 `format_string.slt` records that differ from Spark use an unsuffixed literal such as `3.14159` with `%f`, `%e`, `%g` or `%a`. That makes it the largest category. The cause is the SQL parser, not `format_string`: with the Spark dialect, DataFusion types `3.14159` as DOUBLE and Spark types it as DECIMAL (apache/datafusion#27185). Integer literals have the same mismatch, BIGINT in DataFusion and INT in Spark. Ways to fix it: - **`datafusion.sql_parser.parse_float_as_decimal = true`** already makes DataFusion type `1.5` as a decimal. The sqllogictest runner doesn't set it. With it enabled, 11 of the 245 `.slt` files fail (`try_sum`, `if`, `cast_float_to_timestamp`, `make_interval`, `stack`, `xxhash64`, `floor`, `hypot`, `negative`, `rint`, `format_string`), because their expected results were written with DataFusion's literal types. It doesn't cover integer literals. - **apache/datafusion-sqlparser-rs#2912** parses Spark's numeric literal suffixes (`10Y`, `10S`, `10L`, `1.5D`, `1.5F`, `1.5BD`), which the parser currently reads as column aliases. Once DataFusion maps them to types, tests can write `3.14159D` so that both engines test a DOUBLE. - **Until then**, `3.14159::DOUBLE` makes the tests unambiguous in both engines. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
