andygrove opened a new issue, #3178:
URL: https://github.com/apache/datafusion-comet/issues/3178
## Summary
`array_join` is marked as `Incompatible` in Comet, but the specific
incompatibility is not documented. This issue tracks documenting and
potentially fixing the behavior difference.
## Spark Specification
According to Spark's `array_join` behavior:
- Concatenates array elements into a single string with delimiter
- **Without nullReplacement**: null elements are **ignored** (skipped)
- **With nullReplacement**: null elements are replaced with the specified
string
- Returns null if array or delimiter is null
Examples:
```sql
-- Without null replacement (nulls ignored)
SELECT array_join(array('hello', null, 'world'), ' ');
-- Spark returns: "hello world" (null is skipped)
-- With null replacement
SELECT array_join(array('hello', null, 'world'), ' ', 'NULL');
-- Spark returns: "hello NULL world" (null is replaced)
```
## Current Comet Implementation
Comet uses DataFusion's `array_to_string` function:
```scala
val arrayJoinScalarExpr =
scalarFunctionExprToProto("array_to_string", arrayExprProto,
delimiterExprProto)
```
## Potential Incompatibilities
DataFusion's `array_to_string` may:
1. Include null as empty strings in output (different from skipping)
2. Have different null replacement behavior
3. Handle delimiter differently
## Verification Needed
Test these specific cases:
```sql
-- Null element without replacement
SELECT array_join(array('a', null, 'b'), ',');
-- Spark: "a,b"
-- Comet: ??? (may be "a,,b" or "a,b")
-- Null element with replacement
SELECT array_join(array('a', null, 'b'), ',', 'X');
-- Spark: "a,X,b"
-- Comet: ???
```
---
> **Note:** This issue was generated with AI assistance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
