andygrove opened a new issue, #3152:
URL: https://github.com/apache/datafusion-comet/issues/3152
## What is the problem the feature request solves?
> **Note:** This issue was generated with AI assistance. The specification
details have been extracted from Spark documentation and may need verification.
Comet does not currently support the Spark `array_sort` function, causing
queries using this function to fall back to Spark's JVM execution instead of
running natively on DataFusion.
ArraySort is a higher-order function expression that sorts the elements of
an array using a custom comparator function. It accepts an array and a lambda
function that compares two elements, returning an integer indicating their
relative order.
Supporting this expression would allow more Spark workloads to benefit from
Comet's native acceleration.
## Describe the potential solution
### Spark Specification
**Syntax:**
```sql
array_sort(array, lambda_function)
array_sort(array) -- uses default comparator
```
**Arguments:**
| Argument | Type | Description |
|----------|------|-------------|
| argument | ArrayType | The input array to be sorted |
| function | LambdaFunction | A lambda function that takes two elements and
returns an integer for comparison (negative, zero, or positive) |
| allowNullComparisonResult | Boolean | Internal flag controlling whether
null comparison results are allowed (defaults to legacy configuration) |
**Return Type:** Returns an ArrayType with the same element type and
nullability as the input array.
**Supported Data Types:**
- Input: Any ArrayType containing elements of any data type
- Lambda function must return IntegerType for comparison results
- Supports arrays with null elements when containsNull is true
**Edge Cases:**
- **Null arrays**: Returns null if the input array is null
- **Null elements**: Supported when the array type allows null elements
- **Null comparison results**: Throws
QueryExecutionErrors.comparatorReturnsNull when allowNullComparisonResult is
false and comparator returns null
- **Empty arrays**: Returns empty array unchanged
- **NullType elements**: Arrays with NullType elements are returned unsorted
- **Default comparator**: When no lambda function provided, uses
ArraySort.defaultComparator
**Examples:**
```sql
-- Sort array in ascending order using custom comparator
SELECT array_sort(array(3, 1, 4, 1, 5), (left, right) ->
CASE WHEN left < right THEN -1
WHEN left > right THEN 1
ELSE 0 END);
-- Sort array in descending order
SELECT array_sort(array('c', 'a', 'b'), (left, right) ->
CASE WHEN left > right THEN -1
WHEN left < right THEN 1
ELSE 0 END);
```
```scala
// DataFrame API usage
import org.apache.spark.sql.functions._
df.select(
array_sort(
col("array_column"),
(left, right) => when(left < right, -1)
.when(left > right, 1)
.otherwise(0)
)
)
```
### Implementation Approach
See the [Comet guide on adding new
expressions](https://datafusion.apache.org/comet/contributor-guide/adding_a_new_expression.html)
for detailed instructions.
1. **Scala Serde**: Add expression handler in
`spark/src/main/scala/org/apache/comet/serde/`
2. **Register**: Add to appropriate map in `QueryPlanSerde.scala`
3. **Protobuf**: Add message type in `native/proto/src/proto/expr.proto` if
needed
4. **Rust**: Implement in `native/spark-expr/src/` (check if DataFusion has
built-in support first)
## Additional context
**Difficulty:** Large
**Spark Expression Class:**
`org.apache.spark.sql.catalyst.expressions.ArraySort`
**Related:**
- ArrayBasedSimpleHigherOrderFunction (parent class)
- LambdaFunction (for lambda expression handling)
- Other array functions: array_min, array_max, sort_array
---
*This issue was auto-generated from Spark reference documentation.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
