tlfua commented on PR #7457:
URL: 
https://github.com/apache/datafusion-comet/pull/7457#issuecomment-6942360326

   @andygrove thanks for the review, and @coderfender for the concrete shape. 
With ANSI off the check is now narrowed as you both suggested ( 
https://github.com/apache/datafusion-comet/pull/7457/commits/201edc6cccfb38f056f57aafade386813bb4dcf2
 ):
   
   - Under `spark.sql.ansi.enabled=true` nothing changes: a later key outside 
the `cannotFail` allow-list falls back.
   - With ANSI off a TopK falls back only when a later key can raise in any 
mode: `element_at` on an array with an index that may be 0, `rlike` with a 
non-constant pattern, a user-defined function, `raise_error`, or a cast to or 
from a timestamp or from a date to a string. Keys like `date_add`, `lpad` and 
`array_contains` run natively again, as in 1.2. Since 1.2 ran every TopK 
natively, this only adds safety over the release.
   - `operators.md` now describes both modes, and the ANSI-off section of 
`topk_later_key_ansi.sql` covers `date_add(dt, 1)` and `lpad(s, 5)` staying 
native and `element_at(arr, 0)` falling back.
   
   On plan stability: the PR does not change any approved plan. `q79`'s later 
sort keys are `substr(s_city, 1, 30)` and an aggregate column, both of which 
`cannotFail` accepts, so its shared golden plan keeps 
`CometTakeOrderedAndProject` in every mode. The PR already has 
`run-all-spark-profiles` and `run-spark-3.5-tests`, so the Spark 3.5 plan 
stability suite will report on this commit once the workflows are approved. 
Fork CI: https://github.com/tlfua/datafusion-comet/actions/runs/44025342119
   
   The gaps that are left (the list is best-effort, and the ANSI check reads 
the session flag rather than each expression's eval mode) are noted on 
https://github.com/apache/datafusion-comet/issues/7585 for a follow-up.
   
   @coderfender could you approve the workflows for the new push when you have 
a moment?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to