LiaCastaneda opened a new pull request, #27099:
URL: https://github.com/apache/datafusion/pull/27099
## Which issue does this PR close?
- Closes #14632.
## Rationale for this change
A struct literal combined with a struct column fails once the plan goes
through Substrait. Three forms of it:
```sql
-- s: List<Struct{a, b}>
SELECT coalesce(s, []) FROM t;
SELECT coalesce(s, [named_struct('a', 0, 'b', 'z')]) FROM t;
-- r: Struct{last_used_date, region}, padded with NULL in the other input
SELECT r FROM roles UNION ALL SELECT NULL AS r FROM users;
```
Consuming their Substrait plans fails with:
```
Function 'coalesce' user-defined coercion failed with: ... No function
matches the given name and argument types 'coalesce(List(Struct("a": Utf8, "b":
Utf8)), List(Struct("c0": Utf8, "c1": Utf8)))'
```
```
Unsupported CAST from Struct("c0": Timestamp(µs), "c1": Utf8) to
Struct("last_used_date": Timestamp(µs), "region": Utf8)
```
This happens because Substrait types carry no struct field names: only
relation schemas do. A struct literal (an empty list, a NULL, a constant
struct) is consumed with generated names `c0, c1, ...`, while the column keeps
its real names from the table schema. DataFusion matches struct fields by name,
so it can't unify or cast between the two, even though they have the same shape.
## What changes are included in this PR?
The consumer gives a struct-typed literal the field names of the same-shaped
struct it is combined with, at every nesting level. Substrait matches struct
fields by position, so this keeps the meaning of the plan. Only names and field
metadata change: values, inner types and nullability stay as they are, and
coercion still handles any other difference.
It applies where a literal is combined with other expressions:
- the arguments of a function call (`coalesce`, `nvl`, comparisons, ...)
- the results of a `CASE`
- the values of an `IN` list
- the columns of a set operation (`UNION`, `INTERSECT`, `EXCEPT`), where one
input produces a column with a literal and another reads it with its names
## What is the testing strategy for this PR?
- `consumer_integration.rs`: plans with inline data that are executed and
their rows snapshotted:
- `transform(coalesce(zones, []), x -> ...)` reading a field and a nested
field of the lambda parameter
- `UNION ALL` with a typed NULL struct in either input
- `logical_plans.rs`: `coalesce` of a list of structs with an empty list,
checking that the output type keeps every field name.
- `roundtrip_logical_plan.rs`: SQL -> Substrait -> DataFusion round trips
for `coalesce` with a struct list literal, the `CASE ... ELSE NULL` query from
#14632, `UNION ALL` with `NULL` in either input, and an `IN` list of struct
literals.
Without the change, all of these fail with the errors above. The rest of the
Substrait tests still pass.
## Are there any user-facing changes?
Substrait plans that combine struct literals with struct columns now load
and run. No API changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
