andygrove opened a new issue, #6158: URL: https://github.com/apache/datafusion-comet/issues/6158
### Describe the bug `QueryPlanSerde.supportedSortType` only type-checks a sort that has a single sort order. A multi-column sort whose key includes a string with a non-default collation (Spark 4.0+) therefore converts to `CometSortExec`, which sorts it by raw bytes rather than by the collation. The collation check in `CometShuffleExchangeExec.columnarShuffleFailureReasons` keeps the stage off Comet for hash and range partitioning, which #6110 found is the only thing stopping this there. `SinglePartition` and round-robin exchanges have no such check, so the sort still reaches native. ### Steps to reproduce On `main` at cccc08b7c, default Spark 4.1 profile: ```sql CREATE TABLE repro_coll_sort (_1 INT, _2 STRING) USING parquet; INSERT INTO repro_coll_sort VALUES (1, 'b'), (2, 'A'), (3, 'a'), (4, 'B'); SELECT _1, row_number() OVER (ORDER BY c, _1) AS rn FROM (SELECT _1, _2 COLLATE UTF8_LCASE AS c FROM repro_coll_sort); ``` The executed plan has `CometWindowExec` over `CometSort [c ASC NULLS FIRST, _1 ASC NULLS FIRST]` over a `SinglePartition` `CometColumnarExchange`, and the row numbers differ: | `_1` | Spark `rn` | Comet `rn` | |---|---|---| | 1 | 3 | 4 | | 2 | 1 | 1 | | 3 | 2 | 3 | | 4 | 4 | 2 | Comet orders the rows `A, B, a, b`, which is byte order. Spark orders them `A, a, b, B`, which is `UTF8_LCASE` order with `_1` breaking the ties. ### Expected behavior `CometSortExec` should decline a sort key that contains a non-`UTF8_BINARY` collated string at any position or nesting depth, or route it through the codegen dispatcher, so the result matches Spark. ### Additional context Found while reviewing #6110. There, removing the shuffle collation check makes `listagg(DISTINCT ...)` under `utf8_lcase` return `aabb` instead of `ab`, because `CometSort` stays on a two-column collated key. The #5302 author reported the same limitation, including the window order-spec variant. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
