sunchao opened a new pull request, #5972:
URL: https://github.com/apache/datafusion-comet/pull/5972
## Which issue does this PR close?
Follow-up to #5591 and #5720.
## Rationale for this change
Running an expression through Spark's generated evaluator does not always
preserve how the enclosing Spark operator evaluates it. In particular,
interpreted three-argument `levenshtein` treats a NULL threshold as zero, while
generated evaluation returns NULL. For rows `('', '', NULL)`, `('a', 'b',
NULL)`, and `('a', 'b', 1)`, `collect_list(levenshtein(...))` must retain `[0,
-1, 1]`; generating its arguments separately drops the first two values. The
same distinction affects `NO_CODEGEN`, `array_compact`, and
`approx_count_distinct`.
ANSI `next_day` can also raise errors for rows or operands Spark skips. A
nullable comparison, a LIMIT, or a first-match join can avoid an invalid
weekday. Whole-stage execution can defer a Project or grouped-aggregate result
until a filter or conditional consumes it, and `try_add` can catch an error
raised inside that deferred expression. Materializing it in a Comet batch
changes those results. Separately, a foldable weekday can fail during code
generation where Spark's projection would recover by interpreting a NULL date.
## What changes are included in this PR?
- Preserve interpreted Levenshtein arguments in the affected execution modes
and aggregate/array paths. Safe nonnullable thresholds and generated aggregate
arguments remain eligible for acceleration.
- Check the complete expression tree before an enclosing dispatcher can hide
a NextDay generation failure or a nullable-threshold Levenshtein.
- Preserve ANSI NextDay's nullable-parent, aggregate-argument, LIMIT,
first-match join, and deferred-result boundaries. Retain native execution for
eager inputs, blocking input stages, non-ANSI expressions, and offset-only
collection.
- Adapt the LIMIT/window/AQE buffer handling from #5533 for NextDay. Protect
incompatible aggregate buffers before AQE can remove a sort, and preserve Spark
joins when forced native join rewriting would otherwise replace them.
- Add focused SQL regressions and planner assertions, including
consumed-error controls, native-admission controls, AQE reuse, and TryEval.
Existing feature benchmarks and ordinary collation tests remain in place.
## How are these changes tested?
- Spark **4.0.4** root reactor: **51 tests passed, 0 failed; BUILD
SUCCESS**. Runs the full `CometExecRuleSuite` and all `next_day` and
`levenshtein` SQL fixtures. Production/test compilation, Spotless, Scalastyle,
and `git diff --check` passed.
- Eight paired runtime controls reproduced the mismatches using the base
implementations from `4c2ab9686` and matched Spark with the patched
implementations. The controls cover nullable comparison, deferred Filter, both
NextDay projection modes, Levenshtein NO_CODEGEN, array compact, and both
imperative aggregates. Class origins were checked, and both runs used the same
native library.
- Native-backed tests used the official Linux artifact from [Apache CI run
35014298902](https://github.com/apache/datafusion-comet/actions/runs/35014298902),
commit `347d8cf3d`. Its native tree and build inputs exactly match base
`4c2ab9686`; the intervening commit changes documentation only. The artifact
digest was verified. A local native rebuild was blocked because the configured
registry lacks the pinned DataFusion 55.1.0 release.
- Independent review of the serializer and operator adaptations found no
remaining actionable issues. Other Spark profiles were not run locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
