pingzh opened a new pull request, #7117: URL: https://github.com/apache/datafusion-comet/pull/7117
## Which issue does this PR close? Closes #6741. ## Rationale for this change `make_date(6000000, 1, 1)` is a valid Java calendar date whose epoch day exceeds Date32. Spark raises `ARITHMETIC_OVERFLOW` in either ANSI mode; Comet previously returned NULL or reported an invalid calendar field. Correcting that error must also preserve Spark's year → month → day NULL checks and avoid evaluating rows that lazy consumers skip. ## What changes are included in this PR? - Separate Date32 overflow from invalid calendar fields, and avoid validating scalar arguments on empty input batches. - Use native NULL short-circuiting for computed arguments. Retain Spark evaluation when shared subexpressions or an enclosing expression cannot preserve Spark's evaluation behavior. - Keep affected row pipelines in Spark below LIMIT, filters, sampling, and joins, and for date expressions inside aggregates and ordered limits. Preserve materialization boundaries and reuse the existing broadcast converter when restoring a Spark join. ## How are these changes tested? - Rust regressions cover Date32 boundaries and overflow, invalid-field precedence, NULLs, and empty/nonempty batches with scalar arguments. - SQL regressions cover both ANSI modes, computed NULL-masked arguments, conditional parents, and shared-subexpression fallback. - Planner regressions cover lazy consumers, repeated rule application, materialization boundaries, and reused/AQE broadcast inputs. Execution regressions cover LIMIT and broadcast conversion/cardinality. - Rust formatting, Spotless, and whitespace checks pass. The full native/JVM build (`make`, default Spark 4.1 profile, with JVM tests skipped) passes, including JVM test-source compilation. No local tests were run before opening this draft; CI validation is pending. The planner/serde changes also need the `run-spark-4.1-tests` label for Spark's SQL suite. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
