tlfua commented on code in PR #7457: URL: https://github.com/apache/datafusion-comet/pull/7457#discussion_r4597052758
########## docs/source/user-guide/latest/compatibility/operators.md: ########## @@ -94,6 +94,14 @@ TopK, and ranking window order keys run natively under both default and nondefau searches boundaries over a nullable nested key still runs on Spark ([#6477](https://github.com/apache/datafusion-comet/issues/6477)). +Spark's TopK (`TakeOrderedAndProject`, an `ORDER BY` with a `LIMIT`) evaluates a sort key after +the first only for the rows whose earlier keys tie. Comet evaluates every key over the whole batch, +so under `spark.sql.ansi.enabled=true` a later key such as `b + 1` could raise +`ARITHMETIC_OVERFLOW` on a row that Spark never evaluates it for. A TopK whose later sort key can +fail therefore runs on Spark, unless +`spark.comet.operator.TakeOrderedAndProjectExec.allowIncompatible=true` +([#7246](https://github.com/apache/datafusion-comet/issues/7246)). Review Comment: Update: following @andygrove's and @coderfender's review, the check now depends on ANSI ( https://github.com/apache/datafusion-comet/pull/7457/commits/201edc6cccfb38f056f57aafade386813bb4dcf2 ). Under ANSI it is unchanged; with ANSI off a TopK falls back only when a later key can raise in any mode, so keys like `date_add` and `lpad` run natively again. `operators.md` now describes both modes, and the whitelist above applies only under ANSI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
