andygrove opened a new issue, #307:
URL: https://github.com/apache/datafusion-spark/issues/307

   ### Describe the bug
   
   `date_diff` only works when the optimizer rewrites it. `simplify` 
([datetime/date_diff.rs#L121](https://github.com/apache/datafusion-spark/blob/21ab12776f55d5646431d81f76b55f6bd0b3ea1e/crates/spark/src/function/datetime/date_diff.rs#L121))
 replaces it with a `Date32` subtraction cast to `Int32`, and 
`invoke_with_args` 
([datetime/date_diff.rs#L115](https://github.com/apache/datafusion-spark/blob/21ab12776f55d5646431d81f76b55f6bd0b3ea1e/crates/spark/src/function/datetime/date_diff.rs#L115))
 always returns an error. Any caller that evaluates the function without 
running `simplify` gets that error instead of a result.
   
   Every function needs a working `invoke_with_args`. `simplify` is an optional 
optimization on top of it.
   
   ### To Reproduce
   
   Build a session with the `simplify_expressions` optimizer rule removed, so 
the expression is not constant-folded and `ScalarUDFImpl::simplify` is never 
called. This is how engines that build physical expressions directly, such as 
Comet, run the function.
   
   ```rust
   let rules = Optimizer::new()
       .rules
       .into_iter()
       .filter(|rule| rule.name() != "simplify_expressions")
       .collect();
   let state = SessionStateBuilder::new()
       .with_default_features()
       .with_spark_features()
       .with_optimizer_rules(rules)
       .build();
   ```
   
   ```sql
   SELECT date_diff('2009-07-31'::date, '2009-07-30'::date);
   -- Internal error: Apache Spark `date_diff` should have been simplified to 
standard subtraction.
   ```
   
   ### Expected behavior
   
   `invoke_with_args` computes the same result as the rewritten expression, for 
scalar and array arguments. `simplify` can stay as an optimization.
   
   ### Additional context
   
   Found by running the `.slt` files with `simplify_expressions` disabled. The 
new `evaluate` mode of the sqllogictest runner does this, and it lists 
`datetime/date_diff.slt` as a known failure until this is fixed.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to