andygrove opened a new issue, #216:
URL: https://github.com/apache/datafusion-spark/issues/216
Part of #178 and apache/datafusion-comet#7300.
### What the function does
`array_join(array, delimiter[, nullReplacement])` joins the elements of a
string array with `delimiter`. With
no `nullReplacement`, NULL elements are skipped and no delimiter is written
for them. With one, every NULL
element is written as `nullReplacement`. The result is a STRING (in 4.0+,
the collation of the array's element
type).
The result is NULL if `array` or `delimiter` is NULL. It is also NULL if
`nullReplacement` is given and NULL,
even when the array has no NULL elements. Arguments are evaluated in order
and evaluation stops at the first
NULL: array, then delimiter, then replacement. The result is nullable if any
argument is nullable.
`array` must be `ARRAY<STRING>`. A dedicated coercion rule in
`TypeCoercionHelper` casts any other `ARRAY<T>` to
`ARRAY<STRING>` (keeping `containsNull`) when `T` can be implicitly cast to
STRING. `delimiter` and
`nullReplacement` use `ExpectsInputTypes` and are not cast, so
`array_join(x, 1)` fails with
`DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE` (requiredType `"STRING"`).
The function exists since Spark 2.4.0 and has no aliases.
### Examples
```sql
SELECT array_join(array('hello', 'world'), ' ');
-- hello world
SELECT array_join(array('hello', null ,'world'), ' ');
-- hello world
SELECT array_join(array('hello', null ,'world'), ' ', ',');
-- hello , world
SELECT array_join(array(2.0D, 1.0D), ', ');
-- 2.0, 1.0
SELECT array_join(array(true, false), ', ');
-- true, false
SELECT array_join(array(timestamp '2016-11-15 20:54:00.000', timestamp
'2016-11-12 20:54:00.000'), ', ');
-- 2016-11-15 20:54:00, 2016-11-12 20:54:00
SELECT array_join(array(9223372036854775809, 9223372036854775808), ', ');
-- 9223372036854775809, 9223372036854775808
```
From `CollectionExpressionsSuite`, with delimiter `','`, first without
`nullReplacement` and then with
`nullReplacement` `'NULL'`:
| input | no replacement | replacement `'NULL'` |
|-------------------|----------------|----------------------|
| `['a','b']` | `a,b` | `a,b` |
| `['a',null,'b']` | `a,b` | `a,NULL,b` |
| `[null]` | `''` | `NULL` |
| `['a','b',null]` | `a,b` | `a,b,NULL` |
| `[null,'a','b']` | `a,b` | `NULL,a,b` |
| `[]` | `''` | `''` |
The following inputs return NULL:
- `array_join(NULL, ',')`
- `array_join(array(NULL), NULL)`
- `array_join(array(NULL), ',', NULL)`
### Edge cases to test
- A NULL array, a NULL delimiter, and a NULL `nullReplacement` each give
NULL. A NULL `nullReplacement` gives
NULL even for `array('a')`, which has no NULL elements.
- An empty array gives `''`, not NULL. An array of only NULLs gives `''`
without a replacement.
- Leading, trailing and consecutive NULLs, with and without a replacement.
No delimiter may be emitted for a
skipped NULL.
- An empty delimiter, a multi-character delimiter, an empty
`nullReplacement`, and multi-byte UTF-8 delimiters.
- Non-string element types go through Spark's CAST to STRING, so the output
must match Spark's formatting:
- DOUBLE/FLOAT print as `2.0` and `1.0E10`, NaN as `NaN`, and Infinity as
`Infinity`.
- DECIMAL keeps its scale.
- DATE and TIMESTAMP output depends on `spark.sql.session.timeZone`.
- BINARY elements are cast to STRING.
- Nested arrays (`ARRAY<ARRAY<INT>>`) fail analysis with
`DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE`. Neither
`TypeCoercion` nor `AnsiTypeCoercion` implicitly casts an ARRAY element to
STRING, so the coercion rule leaves
the argument unchanged. DataFusion's `array_to_string` instead flattens
nested lists recursively, and the
kernel must not inherit that.
- `delimiter` and `nullReplacement` may each be a literal or a column. A
per-row NULL in a column gives NULL
only for that row.
- Collations (4.0+): the result type carries the element type's collation,
and trim collations are accepted.
### Differences across Spark versions
- 3.4 and 3.5: the input types are `ArrayType(StringType)` and `StringType`,
and the result is a plain `StringType`.
- 4.0: the input types become
`AbstractArrayType(StringTypeWithCollation(supportsTrimCollation = true))`, and
`dataType` is the array's element string type, so the collation is
propagated.
- 4.1: adds `contextIndependentFoldable` (planner-only).
- 4.2: fixes a codegen bug where a non-nullable array and delimiter with a
nullable `nullReplacement` returned
NULL. The interpreted path was already correct. 4.2 also marks the
expression `expensive`.
- The result semantics are otherwise unchanged.
### Implementation notes
- Spark implements this in
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala`
(`ArrayJoin`). Coercion lives in
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionHelper.scala`.
- DataFusion core's `array_to_string` (aliases `array_join`, `list_join`,
`list_to_string`) is in
`datafusion/functions-nested/src/string.rs`. It differs from Spark as
follows:
- A NULL `null_string` means "skip NULL elements" instead of returning
NULL (apache/datafusion-comet#3178).
- It recurses into nested lists and flattens them.
- It formats floats with Rust `{}`, so `2.0` prints as `2`. Other
non-primitive types go through the Arrow
cast to `Utf8`, whose timestamp and decimal formatting is not Spark's.
- It always returns `Utf8`.
- The empty-array, NULL-array, NULL-delimiter and NULL-element-skipping
behaviour already matches Spark.
- A Spark version can do the element-to-string conversion in `coerce_types`
with Spark's cast (see
`crates/spark/src/function/conversion/cast.rs`), and then either:
- call `array_to_string` after a NULL check on the replacement, or
- use a small dedicated kernel over `ARRAY<STRING>`, which avoids the
recursive flattening and makes nested
arrays a planning error.
- The function belongs in `crates/spark/src/function/array/`. Add
`crates/sqllogictest/test_files/array/array_join.slt` and document it with
`#[user_doc]`.
---
_This issue was generated by an LLM (Claude) from the Apache Spark source
code and tests. Examples come from Spark's `ExpressionDescription` examples,
test suites and `.sql.out` golden files, but please verify them against Spark
before relying on them._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]