tlfua commented on PR #7457: URL: https://github.com/apache/datafusion-comet/pull/7457#issuecomment-6942360326
@andygrove thanks for the review, and @coderfender for the concrete shape. With ANSI off the check is now narrowed as you both suggested ( https://github.com/apache/datafusion-comet/pull/7457/commits/201edc6cccfb38f056f57aafade386813bb4dcf2 ): - Under `spark.sql.ansi.enabled=true` nothing changes: a later key outside the `cannotFail` allow-list falls back. - With ANSI off a TopK falls back only when a later key can raise in any mode: `element_at` on an array with an index that may be 0, `rlike` with a non-constant pattern, a user-defined function, `raise_error`, or a cast to or from a timestamp or from a date to a string. Keys like `date_add`, `lpad` and `array_contains` run natively again, as in 1.2. Since 1.2 ran every TopK natively, this only adds safety over the release. - `operators.md` now describes both modes, and the ANSI-off section of `topk_later_key_ansi.sql` covers `date_add(dt, 1)` and `lpad(s, 5)` staying native and `element_at(arr, 0)` falling back. On plan stability: the PR does not change any approved plan. `q79`'s later sort keys are `substr(s_city, 1, 30)` and an aggregate column, both of which `cannotFail` accepts, so its shared golden plan keeps `CometTakeOrderedAndProject` in every mode. The PR already has `run-all-spark-profiles` and `run-spark-3.5-tests`, so the Spark 3.5 plan stability suite will report on this commit once the workflows are approved. Fork CI: https://github.com/tlfua/datafusion-comet/actions/runs/44025342119 The gaps that are left (the list is best-effort, and the ANSI check reads the session flag rather than each expression's eval mode) are noted on https://github.com/apache/datafusion-comet/issues/7585 for a follow-up. @coderfender could you approve the workflows for the new push when you have a moment? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
