andygrove commented on code in PR #27213:
URL: https://github.com/apache/datafusion/pull/27213#discussion_r4541751595
##########
datafusion/sql/src/expr/value.rs:
##########
@@ -84,6 +85,37 @@ impl<S: ContextProvider> SqlToRel<'_, S> {
&signed_number
};
+ if self.options.dialect == datafusion_common::config::Dialect::Spark {
+ if long {
+ return signed_number.parse::<i64>().map(lit).map_err(|_| {
+ datafusion_common::plan_datafusion_err!(
+ "BIGINT literal {signed_number} is out of range"
+ )
+ });
+ }
+ if let Ok(n) = signed_number.parse::<i64>() {
+ return Ok(match i32::try_from(n) {
+ Ok(n) => lit(n),
Review Comment:
This is the literal that `GROUP BY 1` now passes to
`resolve_positions_to_exprs` (`datafusion/sql/src/utils.rs:243`), which only
takes `Int64` as a position. Could that match any integer literal, with a
Spark-dialect test for `GROUP BY 1`?
##########
datafusion/sql/src/values.rs:
##########
@@ -51,9 +53,44 @@ impl<S: ContextProvider> SqlToRel<'_, S> {
let schema = planner_context.table_schema().unwrap_or(empty_schema);
if schema.fields().is_empty() {
+ if self.options.dialect == Dialect::Spark {
+ coerce_spark_decimal_float_values(&mut values, &schema)?;
+ }
LogicalPlanBuilder::values(values)?.build()
} else {
LogicalPlanBuilder::values_with_schema(values, &schema)?.build()
}
}
}
+
+/// Spark widens a DECIMAL mixed with FLOAT or DOUBLE to DOUBLE. Normalize
these
Review Comment:
This covers `VALUES`, but comparisons, `IF`, `CASE`, `COALESCE` and `IN`
still cast the DOUBLE to `Decimal128(30, 15)`. For example, `SELECT CAST(1e-20
AS DOUBLE) > 0.0` returns false, and `WHERE x > 0.5` fails when `x` holds 1e20
or NaN (repros in the review body). If the coercion rules use DOUBLE for a
DECIMAL mixed with a FLOAT or DOUBLE in the Spark dialect, this function isn't
needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
