tlfua commented on issue #7585: URL: https://github.com/apache/datafusion-comet/issues/7585#issuecomment-6942355561
One more input for this consolidation, from the review of https://github.com/apache/datafusion-comet/pull/7457 ( https://github.com/apache/datafusion-comet/pull/7457#pullrequestreview-5893863590 and https://github.com/apache/datafusion-comet/pull/7457#discussion_r4571919828 ). With `spark.sql.ansi.enabled=false`, #7457 now falls back a TopK only when a later sort key can raise in any mode (`QueryPlanSerde.raisesInAnyMode`). That list is intentionally best-effort. Two gaps are left for a follow-up: 1. **The list is incomplete.** Spark raises regardless of ANSI for, among others, `regexp_extract` with an invalid group index, `slice` with a start of 0 or a negative length, `sequence` and `array_repeat` past the array size limit, `str_to_map` with a duplicate key under the default `mapKeyDedupPolicy`, and `make_ym_interval` on overflow. Comet 1.2 ran all of these natively too, so this is no regression, but a single shared list would cover them for TopK as well as for the other callers above. 2. **The ANSI check reads the session flag, not each expression's eval mode.** A plan analyzed with ANSI on, such as a view that captured `spark.sql.ansi.enabled=true` or a DataFrame analyzed before the session flag changed, keeps ANSI `Cast` and arithmetic nodes. When it runs in a session with ANSI off, the TopK check takes the ANSI-off branch, so such a key can still raise on a row Spark's ordering skips. Asking each node for its own eval mode, as `cannotFail` already does for the arithmetic arms, would close this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
