andygrove opened a new issue, #269:
URL: https://github.com/apache/datafusion-spark/issues/269

   Part of #178. Split out of #145, which was closed as a duplicate of #218 
(`sort_array`).
   
   ### What the function does
   
   `array_sort(array[, func])` sorts an array. Without `func` it uses Spark's 
default comparator, which sorts ascending and
   puts NULL elements **last**. This differs from `sort_array`, which puts 
NULLs first when ascending. The result type is the
   input type. With `func`, the lambda `(left, right) -> int` is the 
comparator. It must return INT, or analysis fails with
   `DATATYPE_MISMATCH.UNEXPECTED_RETURN_TYPE`. If it returns NULL, the query 
fails unless
   `spark.sql.legacy.allowNullComparisonResultInArraySort` (internal, default 
`false`) is set, in which case NULL counts as 0.
   
   Spark implements this as `ArraySort` in 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala`:
   - It is a higher-order function even without `func`. 
`ArraySort.defaultComparator` is the lambda
     `if(left IS NULL AND right IS NULL, 0, if(left IS NULL, 1, if(right IS 
NULL, -1, if(left < right, -1, if(left > right, 1, 0)))))`.
   - Only top-level NULL elements are placed last. Inside struct and array 
elements, `<` and `>` order NULLs below non-null values.
   - Floats compare with `<` and `>`, so NaN is greater than every non-NaN 
value, all NaNs are equal, and `-0.0` equals `0.0`.
   - It sorts with `java.util.Arrays.sort(Object[], Comparator)`, which is 
stable, so equal elements keep their input order.
     An array of `NullType` elements is returned unchanged.
   
   ### Examples
   
   ```sql
   SELECT array_sort(array('b', 'd', null, 'c', 'a'));
   -- ["a","b","c","d",null]
   SELECT array_sort(array(5, 6, 1), (left, right) -> case when left < right 
then -1 when left > right then 1 else 0 end);
   -- [1,5,6]
   SELECT array_sort(array(3, 1, 2), 1);
   -- [3,1,2]   (a constant comparator; the stable sort keeps the input order)
   SELECT array_sort(array(3, 1, 2), map('compare', 0));
   -- error DATATYPE_MISMATCH.UNEXPECTED_RETURN_TYPE
   ```
   
   ### Comet implementation
   
   Comet implements the default-comparator form (`SparkArraySort`, registered 
as `spark_array_sort`). It shares the
   float-ordering and stable-sort code with `sort_array`:
   - 
[`array_funcs/sort_array.rs`](https://github.com/apache/datafusion-comet/blob/3878424d90c23de3c6355a53b293c22822876944/native/spark-expr/src/array_funcs/sort_array.rs)
   
   ### Edge cases to test
   
   - NULL array, empty array, an array of only NULLs, and `array(NULL, NULL)` 
(element type `VOID`).
   - NULLs at the start, middle and end of the input. All of them go last.
   - Struct and array elements that hold NULLs. Those NULLs sort first within 
the element ordering.
   - NaN (with different bit patterns), `±Infinity` and signed zeros. `[0.0, 
-0.0]` and `[-0.0, 0.0]` keep their order.
   - Stability for equal elements, for example structs whose fields compare 
equal.
   
   ### Differences across Spark versions
   
   - 3.4 and 3.5 report `paramIndex` `"1"` for a non-array argument; 4.0 and 
4.1 report it as an ordinal (`"first"`).
   - No other differences in `ArraySort` between 3.4.3, 3.5.8, 4.0.1 and 4.1.1.
   
   ### Implementation notes
   
   - The lambda form depends on lambda support. The higher-order functions in 
#176 have the same dependency, so start with
     the default comparator.
   - DataFusion core's `array_sort` uses an unstable sort and IEEE 754 total 
order for floats, as #218 describes.
     It can be used with `'ASC', 'NULLS LAST'` only for non-float element types 
with no nested NULLs, if the stability
     difference is acceptable there.
   - The function belongs in `crates/spark/src/function/array/`. Add 
`crates/sqllogictest/test_files/array/array_sort.slt`
     and document it with `#[user_doc]`.
   
   ---
   _This issue was generated by an LLM (Claude Code) from the Apache Spark 
source code, Spark's `.sql.out` golden files and
   Comet's source. Please verify the details against Spark before relying on 
them._
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to