andygrove opened a new issue, #97: URL: https://github.com/apache/datafusion-spark/issues/97
With the Spark dialect, DataFusion types an unsuffixed decimal literal such as `3.14159` as DOUBLE, and an integer literal as BIGINT. Spark types them as DECIMAL, and as INT when the value fits. The SQL planner decides these types, and that is tracked upstream in apache/datafusion#27185. This issue tracks the work in this repository. ### Impact Many expected results in the `.slt` files were written with DataFusion's literal types, so they test a type Spark never produces for that query. The PySpark runner (#54) shows this: - 42 of the 119 `format_string.slt` records that differ from Spark use an unsuffixed literal with `%f`, `%e`, `%g` or `%a`. Spark raises `f != org.apache.spark.sql.types.Decimal` for these, and DataFusion formats a double (see #95). - `xxhash64(1)` hashes a BIGINT in DataFusion and an INT in Spark. ### Options 1. **Make the literal's type explicit** in the tests, as `3.14159::DOUBLE`, so that both engines test the same type. This is mechanical and can be done now. 2. **Enable `datafusion.sql_parser.parse_float_as_decimal`** in the sqllogictest runner (`crates/sqllogictest/tests/sqllogictests.rs`), and possibly in `SessionStateBuilderSpark::with_spark_features()` for Spark-dialect users. A trial run made 11 of 245 files fail: `try_sum`, `if`, `cast_float_to_timestamp`, `make_interval`, `stack`, `xxhash64`, `floor`, `hypot`, `negative`, `rint` and `format_string`. Their expected results would need updating. It does not change integer literals. 3. **Use Spark's literal suffixes** (`3.14159D`, `10Y`, `1.5BD`) once apache/datafusion-sqlparser-rs#2912 lands and DataFusion maps the suffixes to types. Spark reads these the same way. Doing 1 now and 2 together with apache/datafusion#27185 means each expected result changes only once. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
