tlfua opened a new issue, #7470: URL: https://github.com/apache/datafusion-comet/issues/7470
### Describe the bug With `spark.sql.ansi.enabled=true`, a sort that has no exchange beneath it fails in Comet with `ARITHMETIC_OVERFLOW` on data that Spark sorts without error, when a sort key after the first can fail. Spark's `SortExec.createSorter` compares rows with `RowOrdering.create(sortOrder, output)`, which goes through `GenerateOrdering`: only the first key is computed for every row (as the sort prefix), and a later key is evaluated for a pair of rows only when the earlier keys tie. Comet's native sort evaluates every sort key over the whole batch, so a later key that can fail raises on rows Spark never evaluates it for. This is the plain-sort counterpart of the top-K case in https://github.com/apache/datafusion-comet/issues/7246 , which https://github.com/apache/datafusion-comet/pull/7457 handles by falling back when a later TopK key can fail. It was raised in review there. ### Steps to reproduce ```sql SET spark.sql.ansi.enabled=true; SET spark.sql.adaptive.enabled=false; CREATE TABLE ps(a int, b int) USING parquet; INSERT INTO ps SELECT /*+ REPARTITION(1) */ * FROM VALUES (1, 2147483647), (2, 5), (3, 2147483647), (4, 7), (5, 9) AS t(a, b); -- No two rows tie on a, so Spark never evaluates b + 1 SELECT a, b FROM ps SORT BY a, b + 1; SELECT a, b FROM (SELECT * FROM ps LIMIT 10) ORDER BY a, b + 1; ``` Both queries return the five rows in Spark and fail in Comet with `[ARITHMETIC_OVERFLOW] integer overflow` (checked on Spark 4.1 with a `CometSqlFileTestSuite` fixture). Not affected, since Spark evaluates every key for every row as well, so both fail: - a global `ORDER BY a, b + 1` over a multi-partition input (the range exchange computes the keys), - window ordering such as `rank() OVER (ORDER BY a, b + 1)` or `row_number() OVER (PARTITION BY ... ORDER BY a, b + 1)` (the analyzer projects `b + 1` below the window). ### Expected behavior Same result as Spark: the queries return the rows. A scoped fallback like the TopK one in https://github.com/apache/datafusion-comet/pull/7457 (fall back when a later sort key can fail) would be the smallest fix; the helper there can be reused. ### Additional context Found while addressing review on https://github.com/apache/datafusion-comet/pull/7457 . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
