tlfua commented on code in PR #7457:
URL: https://github.com/apache/datafusion-comet/pull/7457#discussion_r4597052758


##########
docs/source/user-guide/latest/compatibility/operators.md:
##########
@@ -94,6 +94,14 @@ TopK, and ranking window order keys run natively under both 
default and nondefau
 searches boundaries over a nullable nested key still runs on Spark
 ([#6477](https://github.com/apache/datafusion-comet/issues/6477)).
 
+Spark's TopK (`TakeOrderedAndProject`, an `ORDER BY` with a `LIMIT`) evaluates 
a sort key after
+the first only for the rows whose earlier keys tie. Comet evaluates every key 
over the whole batch,
+so under `spark.sql.ansi.enabled=true` a later key such as `b + 1` could raise
+`ARITHMETIC_OVERFLOW` on a row that Spark never evaluates it for. A TopK whose 
later sort key can
+fail therefore runs on Spark, unless
+`spark.comet.operator.TakeOrderedAndProjectExec.allowIncompatible=true`
+([#7246](https://github.com/apache/datafusion-comet/issues/7246)).

Review Comment:
   Update: following @andygrove's and @coderfender's review, the check now 
depends on ANSI ( 
https://github.com/apache/datafusion-comet/pull/7457/commits/201edc6cccfb38f056f57aafade386813bb4dcf2
 ). Under ANSI it is unchanged; with ANSI off a TopK falls back only when a 
later key can raise in any mode, so keys like `date_add` and `lpad` run 
natively again. `operators.md` now describes both modes, and the whitelist 
above applies only under ANSI.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to