tlfua commented on PR #7457: URL: https://github.com/apache/datafusion-comet/pull/7457#issuecomment-6961187380
Thanks @andygrove, all five points are addressed in ade66196f102207257bd53fa3bae3e6134cf207e and d4f087ea5f49567e9dbaa32f2eee4dad46c2295a . 1. **Timestamp casts with ANSI off.** Removed the cast arm from `raisesInAnyMode`, so with ANSI off a timestamp-cast later key now runs in Comet. Under ANSI, `CometCast.cannotFail` still rejects these casts. @coderfender, this reverses the arm you added deliberately: Spark does not raise for these casts with ANSI off, and the chrono range failure is a Comet kernel bug that hits a projection the same way, so it belongs to https://github.com/apache/datafusion-comet/issues/7471 rather than to the TopK check. 2. **Views.** With ANSI off, a later key that holds a node still in ANSI eval mode (arithmetic and casts by their `EvalMode`, plus `failOnError`/`ansiEnabled` on `UnaryMinus`, `Abs`, `Round`, `BRound`, `ElementAt`, `GetArrayItem` and `Elt`) and that `cannotFail` rejects now falls back. The test creates `SELECT a, b FROM topk ORDER BY a, b + 1 LIMIT 2` as a temporary view with ANSI on and queries it with ANSI off. Spark 4.1 keeps the arithmetic eval mode inside a `NumericEvalContext`, which the first version missed; fork CI caught it on the Spark 4.1 profile, and the follow-up commit reads `evalMode` by class instead. A view that only projects `b + 1 AS c` and is then sorted by `c` is not affected, because the projection evaluates `b + 1` for every row in Spark too. 3. **`slice`, `array_insert`, `sequence`.** Added to `raisesInAnyMode`: `slice` unless the start is a nonzero constant and the length a non-negative constant, `array_insert` unless the position is a nonzero constant, and `sequence` unless all of its arguments are constants (invalid constant bounds already fail in constant folding). 4. **Tests.** - Timestamp casts: since they no longer fall back with ANSI off, `date_add(ts, 1)` and `concat(s, ts)` are plain `query` tests that check they stay in Comet. - UDF: `raise_error` cannot reach this check in a SQL file test, because Comet has no serde for `RaiseError` and the TopK falls back for that first. So `CometCodegenSuite` has a ScalaUDF later-key test with ANSI off instead. - `element_at(arr, 1)`, `slice(arr, 1, 1)` and `array_insert(arr, 1, 0)` stay in Comet; column-driven `slice`, `array_insert` and `sequence` fall back. 5. **Docs.** `operators.md` now says "runs in Comet", lists the new ANSI-off cases and the view case, says `allowIncompatible` applies in either mode, and links https://github.com/apache/datafusion-comet/issues/7471 for timestamp casts. I also replaced the same "native" wording in the code comments. Fork CI passed on d4f087ea5: https://github.com/tlfua/datafusion-comet/actions/runs/44155798081 . One Spark SQL job was rerun after a Maven Central download failure. Could you approve the workflows again when you have a moment? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
