csbiy opened a new pull request, #3364:
URL: https://github.com/apache/datafusion-comet/pull/3364
## Summary
This PR addresses issue #3175 by documenting the specific null handling
incompatibility between Spark and Comet for the `arrays_overlap` function.
## Changes Made
### 1. Code Documentation (`arrays.scala`)
Updated `CometArraysOverlap.getSupportLevel()` to return
`Incompatible(Some(...))` with detailed explanation and concrete example.
**Before:**
```scala
override def getSupportLevel(expr: ArraysOverlap): SupportLevel =
Incompatible(None)
```
**After:**
```scala
override def getSupportLevel(expr: ArraysOverlap): SupportLevel =
Incompatible(Some(
"Null handling differs from Spark: DataFusion's array_has_any returns
false when no " +
"common elements are found, even if null elements exist. Spark returns
null in such " +
"cases following three-valued logic (SQL standard). Example: " +
"arrays_overlap(array(1, null, 3), array(4, 5)) returns null in Spark
but false in Comet."))
```
### 2. Test Coverage (`CometArrayExpressionSuite.scala`)
Added comprehensive test `arrays_overlap - null handling behavior
verification` with 6 test cases:
1. Common element exists - returns `true`
2. No common elements, no nulls - returns `false`
3. No common elements, null exists - Spark: `null`, Comet: `false`
(documented incompatibility)
4. Common element with null present - returns `true`
5. Both arrays with null, no common elements - behavior documented
6. Empty array cases - edge case covered
### 3. User Documentation (`expressions.md`)
Updated the Array Expressions table with specific explanation and example
showing the three-valued logic difference.
## Root Cause Analysis
### Spark Behavior (Three-Valued Logic)
Spark follows SQL's three-valued logic (true, false, null):
- Returns `true` if common elements found
- Returns `false` if no common elements AND no nulls
- Returns `null` if no common elements BUT nulls exist (indeterminate)
### Comet Behavior
Comet uses DataFusion's `array_has_any` function:
- Returns `true` if common elements found
- Returns `false` in all other cases (no three-valued logic support)
## Example Demonstrating Incompatibility
```sql
SELECT arrays_overlap(array(1, null, 3), array(4, 5))
```
| System | Result | Reason |
|--------|--------|--------|
| Spark | `null` | No common elements, but null exists - indeterminate |
| Comet | `false` | DataFusion doesn't implement three-valued logic |
## Why This Matters
Users who enable `arrays_overlap` with
`spark.comet.expression.ArraysOverlap.allowIncompatible=true` need to
understand:
1. Query results may differ when nulls are present
2. Downstream logic relying on null distinction (vs false) may break
3. JOIN conditions or filters using this function may behave differently
## Testing Notes
Local test execution encountered environment Java version compatibility
issues (unrelated to code changes). Test code is syntactically correct and
follows existing patterns. CI environment should run tests successfully with
proper Java configuration.
## Files Modified
```
spark/src/main/scala/org/apache/comet/serde/arrays.scala
spark/src/test/scala/org/apache/comet/CometArrayExpressionSuite.scala
docs/source/user-guide/latest/expressions.md
```
## Closes
Closes #3175
## Checklist
- [x] Documented specific incompatibility reason in code
- [x] Added test cases verifying behavior
- [x] Updated user-facing documentation
- [x] Followed existing code style and patterns
- [x] Added concrete example for user clarity
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
