andygrove opened a new issue, #3158:
URL: https://github.com/apache/datafusion-comet/issues/3158
## What is the problem the feature request solves?
> **Note:** This issue was generated with AI assistance. The specification
details have been extracted from Spark documentation and may need verification.
Comet does not currently support the Spark `array_for_all` function, causing
queries using this function to fall back to Spark's JVM execution instead of
running natively on DataFusion.
The `ArrayForAll` expression tests whether all elements in an array satisfy
a given predicate condition. It applies a lambda function to each element and
returns true only if the predicate evaluates to true for every non-null element
in the array.
Supporting this expression would allow more Spark workloads to benefit from
Comet's native acceleration.
## Describe the potential solution
### Spark Specification
**Syntax:**
```sql
forall(array_expression, lambda_function)
```
```scala
// DataFrame API usage
df.select(forall(col("array_column"), x => x % 2 === 0))
```
**Arguments:**
| Argument | Type | Description |
|----------|------|-------------|
| array_expression | ArrayType | The input array to evaluate |
| lambda_function | LambdaFunction | A lambda function that takes an array
element and returns a Boolean |
**Return Type:** `BooleanType` - Returns true if all elements satisfy the
condition, false if any element fails the condition, or null under specific
null-handling scenarios.
**Supported Data Types:**
- Input: Any `ArrayType` containing elements of any data type
- Lambda function must return `BooleanType`
- Array elements can be nullable
**Edge Cases:**
- **Null array**: Returns null if the input array itself is null
- **Empty array**: Returns true (vacuous truth - all zero elements satisfy
the condition)
- **Null elements with all true**: Returns null if any element evaluation
returns null AND all non-null evaluations return true
- **Null elements with any false**: Returns false if any element evaluation
returns false, regardless of null presence
- **All null elements**: Returns null if all element evaluations return null
**Examples:**
```sql
-- All elements are even
SELECT forall(array(2, 4, 6), x -> x % 2 = 0);
-- Returns: true
-- Not all elements are even
SELECT forall(array(2, 3, 4), x -> x % 2 = 0);
-- Returns: false
-- Contains null with all non-null elements satisfying condition
SELECT forall(array(2, null, 8), x -> x % 2 = 0);
-- Returns: null
-- Contains null but has false element
SELECT forall(array(1, null, 8), x -> x % 2 = 0);
-- Returns: false
```
```scala
// DataFrame API examples
import org.apache.spark.sql.functions._
// Check if all numbers are positive
df.select(forall(col("numbers"), x => x > 0))
// Check if all strings have length > 3
df.select(forall(col("words"), x => length(x) > 3))
```
### Implementation Approach
See the [Comet guide on adding new
expressions](https://datafusion.apache.org/comet/contributor-guide/adding_a_new_expression.html)
for detailed instructions.
1. **Scala Serde**: Add expression handler in
`spark/src/main/scala/org/apache/comet/serde/`
2. **Register**: Add to appropriate map in `QueryPlanSerde.scala`
3. **Protobuf**: Add message type in `native/proto/src/proto/expr.proto` if
needed
4. **Rust**: Implement in `native/spark-expr/src/` (check if DataFusion has
built-in support first)
## Additional context
**Difficulty:** Medium
**Spark Expression Class:**
`org.apache.spark.sql.catalyst.expressions.ArrayForAll`
**Related:**
- `ArrayExists` - Tests if any element satisfies a condition
- `ArrayFilter` - Filters array elements based on a predicate
- `ArrayTransform` - Transforms array elements using a lambda function
---
*This issue was auto-generated from Spark reference documentation.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
