andygrove opened a new issue, #223:
URL: https://github.com/apache/datafusion-spark/issues/223
Part of #178 and apache/datafusion-comet#7300.
### What the function does
`flatten(arrayOfArrays)` concatenates the inner arrays of an
`ARRAY<ARRAY<T>>` into a single `ARRAY<T>`, in
order. It removes exactly one level of nesting. Deeper levels are left as
they are, so
`ARRAY<ARRAY<ARRAY<T>>>` becomes `ARRAY<ARRAY<T>>`.
The result type is the input's element type (`ARRAY<T>`, with the inner
`containsNull`). Input that is not an
array of arrays fails analysis with
`DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE` (requiredType
`"ARRAY" of "ARRAY"`). There is no implicit cast.
NULL handling:
- A NULL outer array gives NULL.
- If any inner array is NULL, the whole result is NULL.
- NULL elements inside the inner arrays are kept.
- The result is nullable when the input is nullable or the outer
`containsNull` is true.
The function exists since Spark 2.4.0 and has no aliases.
### Examples
```sql
SELECT flatten(array(array(1, 2), array(3, 4)));
-- [1,2,3,4]
```
From `CollectionExpressionsSuite`:
| input | result |
|--------------------------------------------|---------------------------|
| `[[1,2,3],[4,5],[6]]` | `[1,2,3,4,5,6]` |
| `[[],[1,2],[3,4]]` | `[1,2,3,4]` |
| `[[1,2],[],[3,4]]` | `[1,2,3,4]` |
| `[[],[],[]]` | `[]` |
| `[]` | `[]` |
| `[[null,null,null],[4,null]]` | `[null,null,null,4,null]` |
| `[null,[1,2]]` | NULL |
| `[[1,2],null]` | NULL |
| `[null]` | NULL |
| NULL | NULL |
| `[[["a","b"],["c"]],[["d","e"]]]` | `[["a","b"],["c"],["d","e"]]`
|
### Edge cases to test
- A NULL inner array anywhere in the outer array makes the result NULL, even
when the other inner arrays are
non-empty.
- Empty inner arrays and an empty outer array give `[]`, not NULL.
- NULL elements inside inner arrays are kept in place.
- Three-level nesting: only one level is removed.
- Inner element types: string, binary, struct, map, decimal, timestamp, and
nested arrays.
- A total element count above `ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH`
raises
`COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION` in 4.x and
`_LEGACY_ERROR_TEMP_2160` in 3.x. This probably only
needs a unit test.
- A flat array such as `flatten(array(1, 2))` must be a planning error, not
a no-op.
- Column inputs with mixed NULL and non-NULL rows, and sliced list arrays
where offsets do not start at 0.
- Inputs that arrive as Arrow `LargeList` or `FixedSizeList`, at either the
outer or the inner level.
### Differences across Spark versions
- Semantics are unchanged from 3.4 to 4.2.
- 4.0 replaces the `NullIntolerant` trait with a `nullIntolerant` override.
- The size-limit error changes from `_LEGACY_ERROR_TEMP_2160` in 3.x to
`COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION`
in 4.x.
### Implementation notes
- Spark implements this in
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala`
(`Flatten`).
- DataFusion core's `flatten` is in
`datafusion/functions-nested/src/flatten.rs`. It differs from Spark as follows:
- It drops the inner validity buffer and stitches the inner offsets
together. A NULL inner array is therefore
treated as empty, or as whatever values its offset range covers if that
range is non-empty, and the remaining
elements are returned. Spark returns NULL for the whole row.
- A non-nested array (`List<Int32>`) is returned unchanged, where Spark
rejects it at analysis.
- The user doc says "applies to any depth", but the code removes one
level, as Spark does. Test this.
- It reuses the inner list field, so the result's element nullability
follows the inner field, which matches
Spark.
- A Spark version can wrap DataFusion's kernel and add a row mask for rows
where any inner list is NULL. Compute
it from the inner list's null buffer over each outer row's offset range,
then union it with the outer null
buffer. The signature should also reject non-nested input. The NULL-mask
helpers in
`crates/spark/src/function/array/array_contains.rs`
(`compute_row_has_nulls`, `mask_with_list_nulls`) do
similar per-row null work and could be generalized.
- The function belongs in `crates/spark/src/function/array/`. Add
`crates/sqllogictest/test_files/array/flatten.slt` and document it with
`#[user_doc]`.
---
_This issue was generated by an LLM (Claude) from the Apache Spark source
code and tests. Examples come from Spark's `ExpressionDescription` examples,
test suites and `.sql.out` golden files, but please verify them against Spark
before relying on them._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]