kazantsev-maksim commented on PR #4744:
URL:
https://github.com/apache/datafusion-comet/pull/4744#issuecomment-5982544217
Thanks for the thorough investigation and reproduction cases, @sunchao!
I have resolved both remaining blockers:
1. **Nested Complex Captures (`capturesComplexOuterVariable`):**
- The capture inspection now tracks both outer table columns
(`AttributeReference`) and enclosing lambda parameters
(`SparkNamedLambdaVariable`).
- Any complex capture (`ArrayType`, `MapType`, `StructType`) across
nested lambda boundaries now safely degrades to JVM codegen dispatch,
eliminating the quadratic memory replication in `take_arrays`.
2. **Conditionals & Null-Safe Comparisons
(`hasUnsupportedNativeExpressions`):**
- Removed the `BooleanType` restriction. All conditional expressions
(`CaseWhen`, `If`, `Coalesce`), regardless of whether they return booleans or
scalar values, now degrade to JVM codegen dispatch.
- Added `EqualNullSafe` (`<=>`) to the dispatch fallback list.
- Removed the native `coalesce` test from `array_filter.sql` (which runs
strictly with codegen disabled) since it now properly delegates to the codegen
dispatcher.
All native tests and fallback regressions now pass cleanly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
