sunchao commented on PR #5972:
URL:
https://github.com/apache/datafusion-comet/pull/5972#issuecomment-5982822631
Addressed the review in the updated branch, stacked on #5533:
- Enrolled `NextDay` in the shared policy; removed the duplicate traversal,
retained the early exit/configuration and main's shared aggregate restoration.
- Added valid-weekday admission below LIMIT, eager parent/sort admission,
and native unfiltered MAX controls. Serializer and deferred-reference analysis
share their eager-input rule; aggregate fallback is scoped to FILTER and
First/Last.
- Nullable three-argument `levenshtein` now stays in Spark for FALLBACK,
CODEGEN_ONLY and NO_CODEGEN, including containing JVM-dispatched expressions.
Safe forms remain native.
- Generalized whole-expression dispatch, shared the interpreted-mode helper,
restored the raw-bytes explanation, and extended the existing compatibility
section.
The deferred-projection analysis remains in this follow-up to the shared
policy rather than introducing another open dependent PR. Its scope is covered
by six focused planner tests and the nine-test shared mask suite, in addition
to nine next_day and eight levenshtein SQL cases. All 32 focused tests pass on
Spark 4.1.3 across the current-source runs; other profiles and full Spark SQL
suites are requested in CI.
The Spark JIRA filing is still outstanding: the available Jira connector is
not authenticated. I checked current Spark master and the same eval/doGenCode
discrepancy remains. Here is a ready-to-file report so that limitation is
explicit:
**Title:** Nullable levenshtein threshold returns different results in
interpreted and generated projections
Run with Comet disabled, using stored input to avoid constant folding:
```sql
CREATE TABLE levenshtein_null_threshold (s STRING, t STRING, threshold INT)
USING parquet;
INSERT INTO levenshtein_null_threshold VALUES ('same', 'same', NULL), ('a',
'b', NULL);
SET spark.sql.codegen.wholeStage=false;
SET spark.sql.codegen.factoryMode=NO_CODEGEN;
SELECT levenshtein(s, t, threshold) FROM levenshtein_null_threshold;
SET spark.sql.codegen.factoryMode=CODEGEN_ONLY;
SELECT levenshtein(s, t, threshold) FROM levenshtein_null_threshold;
```
`Levenshtein.eval` passes `Some(null)` through `asInstanceOf[Int]`,
producing a zero threshold; generated evaluation checks threshold nullness and
returns NULL. Expected: both paths return NULL. The default FALLBACK mode can
encounter either projection implementation after a compilation failure. The
Comet workaround must remain until upstream behavior is consistent across
supported Spark versions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
