andygrove opened a new issue, #269:
URL: https://github.com/apache/datafusion-spark/issues/269
Part of #178. Split out of #145, which was closed as a duplicate of #218
(`sort_array`).
### What the function does
`array_sort(array[, func])` sorts an array. Without `func` it uses Spark's
default comparator, which sorts ascending and
puts NULL elements **last**. This differs from `sort_array`, which puts
NULLs first when ascending. The result type is the
input type. With `func`, the lambda `(left, right) -> int` is the
comparator. It must return INT, or analysis fails with
`DATATYPE_MISMATCH.UNEXPECTED_RETURN_TYPE`. If it returns NULL, the query
fails unless
`spark.sql.legacy.allowNullComparisonResultInArraySort` (internal, default
`false`) is set, in which case NULL counts as 0.
Spark implements this as `ArraySort` in
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala`:
- It is a higher-order function even without `func`.
`ArraySort.defaultComparator` is the lambda
`if(left IS NULL AND right IS NULL, 0, if(left IS NULL, 1, if(right IS
NULL, -1, if(left < right, -1, if(left > right, 1, 0)))))`.
- Only top-level NULL elements are placed last. Inside struct and array
elements, `<` and `>` order NULLs below non-null values.
- Floats compare with `<` and `>`, so NaN is greater than every non-NaN
value, all NaNs are equal, and `-0.0` equals `0.0`.
- It sorts with `java.util.Arrays.sort(Object[], Comparator)`, which is
stable, so equal elements keep their input order.
An array of `NullType` elements is returned unchanged.
### Examples
```sql
SELECT array_sort(array('b', 'd', null, 'c', 'a'));
-- ["a","b","c","d",null]
SELECT array_sort(array(5, 6, 1), (left, right) -> case when left < right
then -1 when left > right then 1 else 0 end);
-- [1,5,6]
SELECT array_sort(array(3, 1, 2), 1);
-- [3,1,2] (a constant comparator; the stable sort keeps the input order)
SELECT array_sort(array(3, 1, 2), map('compare', 0));
-- error DATATYPE_MISMATCH.UNEXPECTED_RETURN_TYPE
```
### Comet implementation
Comet implements the default-comparator form (`SparkArraySort`, registered
as `spark_array_sort`). It shares the
float-ordering and stable-sort code with `sort_array`:
-
[`array_funcs/sort_array.rs`](https://github.com/apache/datafusion-comet/blob/3878424d90c23de3c6355a53b293c22822876944/native/spark-expr/src/array_funcs/sort_array.rs)
### Edge cases to test
- NULL array, empty array, an array of only NULLs, and `array(NULL, NULL)`
(element type `VOID`).
- NULLs at the start, middle and end of the input. All of them go last.
- Struct and array elements that hold NULLs. Those NULLs sort first within
the element ordering.
- NaN (with different bit patterns), `±Infinity` and signed zeros. `[0.0,
-0.0]` and `[-0.0, 0.0]` keep their order.
- Stability for equal elements, for example structs whose fields compare
equal.
### Differences across Spark versions
- 3.4 and 3.5 report `paramIndex` `"1"` for a non-array argument; 4.0 and
4.1 report it as an ordinal (`"first"`).
- No other differences in `ArraySort` between 3.4.3, 3.5.8, 4.0.1 and 4.1.1.
### Implementation notes
- The lambda form depends on lambda support. The higher-order functions in
#176 have the same dependency, so start with
the default comparator.
- DataFusion core's `array_sort` uses an unstable sort and IEEE 754 total
order for floats, as #218 describes.
It can be used with `'ASC', 'NULLS LAST'` only for non-float element types
with no nested NULLs, if the stability
difference is acceptable there.
- The function belongs in `crates/spark/src/function/array/`. Add
`crates/sqllogictest/test_files/array/array_sort.slt`
and document it with `#[user_doc]`.
---
_This issue was generated by an LLM (Claude Code) from the Apache Spark
source code, Spark's `.sql.out` golden files and
Comet's source. Please verify the details against Spark before relying on
them._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]