SparkApplicationMaster commented on code in PR #16946: URL: https://github.com/apache/datafusion/pull/16946#discussion_r2296299288
########## datafusion/sqllogictest/test_files/spark/conditional/if.slt: ########## @@ -21,7 +21,135 @@ # For more information, please see: # https://github.com/apache/datafusion/issues/15914 -## Original Query: SELECT if(1 < 2, 'a', 'b'); -## PySpark 3.5.5 Result: {'(IF((1 < 2), a, b))': 'a', 'typeof((IF((1 < 2), a, b)))': 'string', 'typeof((1 < 2))': 'boolean', 'typeof(a)': 'string', 'typeof(b)': 'string'} -#query -#SELECT if((1 < 2)::boolean, 'a'::string, 'b'::string); +## Basic IF function tests + +# Test basic true condition +query T +SELECT if(true, 'yes', 'no'); Review Comment: please also check if when(...).otherwise(`else_expr`) used here: https://github.com/apache/datafusion/pull/16946/files#diff-8184a681e2d4b84411030426011f4e80cc4a79e2debd39f6290d0159d83a63a5R97 does not eagerly calculate `else_expr`. because I faced it calculating `else_expr` when using `when` expression: https://github.com/lakehq/sail/issues/648 and the fix was just removing else_expr from when expr and placing it to the last branch of when with true predicate: https://github.com/lakehq/sail/pull/649/files also checked this now by adding in datafusion if.slt: ``` query I SELECT case when true then 1 / 1 else 1 / 0 end; ---- 1 ``` and got this: ``` 1. query failed: DataFusion error: Arrow error: Divide by zero error [SQL] SELECT case when true then 1 / 1 else 1 / 0 end; ``` but this: ``` query I SELECT case when a then 1 / 1 else 1 / b end FROM (VALUES (false, 1), (true, 0)) t(a, b); ---- 1 1 ``` works OK, maybe the problem with example above is only about eager constant evaluation in else_expr, not every expression Not sure if eager evaluation of else expression is OK in datafusion, but in spark it's definitely NOT OK, and there is a doctest checking this: https://github.com/apache/spark/blob/326052ec8280d8bf8ee1904504be3b62a72d3d29/python/pyspark/sql/column.py#L1418-L1427 `raise_error(literal(str))` is also constant expression, so it evaluates when not intented to be -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
