tyrobro commented on PR #301:
URL: https://github.com/apache/datafusion-spark/pull/301#issuecomment-6917310605

   Thanks for the thorough review @andygrove, I have addressed all nine points:
   
   1. Fixed the .slt comments that didn’t match their queries, and reworked the 
DECIMAL/DECIMAL case to actually assert on arrow_typeof so a wrong coercion 
target won’t go unnoticed again.
   2. Unified the f32/f64 kernels into a single zip based implementation 
instead of two near identical per row loops.
   3. Fixed the -0.0 test to assert on 1 / nanvl(-0.0, 1.0) = -Infinity, which 
actually distinguishes it from 0.0.
   4. Fixed STRING + ANSI mode: string arguments now skip automatic 
coerce_types coercion and get a manual CastOptions { safe: !enable_ansi_mode } 
cast in invoke_with_args, so an invalid string returns NULL in legacy mode 
instead of failing at planning.
   5. Added an explicit allow list in coerce_types so BOOLEAN (and anything 
else outside numeric/decimal/string/null) is rejected with a plan_err instead 
of silently accepted.
   6. Fixed the NULL/FLOAT coercion asymmetry: nanvl(NULL, FLOAT) now resolves 
to DOUBLE, nanvl(FLOAT, NULL) stays FLOAT, matching Spark.
   7. Added tests feeding NaN through a column (not just a literal) for both 
DOUBLE and FLOAT, this is what actually exposed point 9 below.
   8. Fixed volatile expr1 (e.g. random()) being evaluated twice by the 
simplify() CASE rewrite now bypasses the rewrite via is_volatile() and falls 
back to eager evaluation.
   9. Fixed the core bug: expr1 != expr1 never fires for real NaN data under 
Arrow’s totalOrder comparison semantics (only appeared to work on 
constant-folded literals). Replaced with isnan(expr1).
   
   One more fix that came out of testing point 4: the simplify() CASE rewrite 
also needed to skip string arguments entirely (a CASE with a Utf8 THEN-branch 
and Float64 ELSE-branch fails at planning), so strings now bypass simplify() 
the same way volatile expressions do.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to