tyrobro commented on PR #301:
URL: https://github.com/apache/datafusion-spark/pull/301#issuecomment-6917310605
Thanks for the thorough review @andygrove, I have addressed all nine points:
1. Fixed the .slt comments that didn’t match their queries, and reworked the
DECIMAL/DECIMAL case to actually assert on arrow_typeof so a wrong coercion
target won’t go unnoticed again.
2. Unified the f32/f64 kernels into a single zip based implementation
instead of two near identical per row loops.
3. Fixed the -0.0 test to assert on 1 / nanvl(-0.0, 1.0) = -Infinity, which
actually distinguishes it from 0.0.
4. Fixed STRING + ANSI mode: string arguments now skip automatic
coerce_types coercion and get a manual CastOptions { safe: !enable_ansi_mode }
cast in invoke_with_args, so an invalid string returns NULL in legacy mode
instead of failing at planning.
5. Added an explicit allow list in coerce_types so BOOLEAN (and anything
else outside numeric/decimal/string/null) is rejected with a plan_err instead
of silently accepted.
6. Fixed the NULL/FLOAT coercion asymmetry: nanvl(NULL, FLOAT) now resolves
to DOUBLE, nanvl(FLOAT, NULL) stays FLOAT, matching Spark.
7. Added tests feeding NaN through a column (not just a literal) for both
DOUBLE and FLOAT, this is what actually exposed point 9 below.
8. Fixed volatile expr1 (e.g. random()) being evaluated twice by the
simplify() CASE rewrite now bypasses the rewrite via is_volatile() and falls
back to eager evaluation.
9. Fixed the core bug: expr1 != expr1 never fires for real NaN data under
Arrow’s totalOrder comparison semantics (only appeared to work on
constant-folded literals). Replaced with isnan(expr1).
One more fix that came out of testing point 4: the simplify() CASE rewrite
also needed to skip string arguments entirely (a CASE with a Utf8 THEN-branch
and Float64 ELSE-branch fails at planning), so strings now bypass simplify()
the same way volatile expressions do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]