TRO-Wolf opened a new issue, #26178:
URL: https://github.com/apache/datafusion/issues/26178
### Describe the bug
A `CAST` over an expression that the optimizer simplifies to a bare column
carrying field metadata raises an internal error when it sits in the branches
of a `UNION ALL` under an aggregate:
```
Internal error: Physical input schema should be the same as the one
converted from logical input schema. Differences:
- field metadata at index 0 [x]: (physical) {"PARQUET:field_id": "1"}
vs (logical) {}.
```
Reproduced on DataFusion 54.1.0 (through `datafusion-python` 54.1.0, Linux
x86_64). Not checked against `main`.
### To Reproduce
```python
import datafusion, pyarrow as pa
schema = pa.schema([
pa.field("b", pa.int64(), nullable=False, metadata={"PARQUET:field_id":
"1"}),
pa.field("n", pa.int64(), metadata={"PARQUET:field_id": "2"}),
])
t = pa.table({"b": [1, 2, 3], "n": [1, None, 3]}, schema=schema)
ctx = datafusion.SessionContext()
ctx.register_record_batches("t", [t.to_batches()])
ctx.sql("""
SELECT max(x) AS m, count(x) AS c FROM
(SELECT CAST(coalesce(b, 7) AS BIGINT) AS x FROM t
UNION ALL
SELECT CAST(coalesce(b, 7) AS BIGINT) AS x FROM t) q
""").collect()
```
`b` is `NOT NULL`, so `coalesce(b, 7)` simplifies to `b`.
Controls on the same build, all of which answer:
| Query | Result |
|---|---|
| the same query without the `UNION ALL` | `m = 3, c = 3` |
| `UNION ALL` over `CAST(b AS BIGINT)` (no `coalesce`) | `m = 3, c = 6` |
| the failing query against the same data registered without field metadata
| `m = 3, c = 6` |
### Expected behavior
`m = 3, c = 6`.
### Additional context
Reading the code, the logical side (`cast_output_field`) and the physical
side (`CastExpr`'s resolved target field) both propagate the child's field for
a default-target cast, so they agree when both are derived from the same plan
state. Here they are not: the `UNION`'s schema was computed before the
`coalesce` was simplified away (child = a function result, no metadata), while
the physical plan is built from the simplified branches (child = the column,
with its metadata).
Two possible directions, offered as a suggestion only:
1. Recompute the union schema after expression simplification. This fixes
the `UNION` shape.
2. Treat a cast's output as a new value that carries none of its child's
field metadata, on both the logical and the physical side. This would fix every
boundary whose schema is fixed before simplification, and it also removes a
visible oddity: today `SELECT CAST(col AS <type>)` returns a field that still
carries the source column's `PARQUET:field_id`.
Possibly related: #19049 (closed; `UNION ALL` plus `optimize_projections`
with Parquet field metadata), #25978 and #25369 (logical/physical mismatch from
a rewrite under an aggregate).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
