alamb opened a new pull request, #24278:
URL: https://github.com/apache/datafusion/pull/24278
## Which issue does this PR close?
- Part of #24069 (reproducer only; the tests fail on `main` by design, so
this does not close the issue).
## Rationale for this change
#24069 reports `GroupedHashAggregateStream::emit` failing with `ArrowError:
column types must match schema types, expected List(Struct("colA": Boolean))
but found List(Struct("colA": non-null Boolean))`. The trigger is input batches
whose data types are *stricter* than the declared schema — a nested struct
field that is non-nullable while the schema declares it nullable. DataFusion
admits such batches: `MemTable::try_new` validates them with
[`Schema::contains`](https://docs.rs/arrow/latest/arrow/datatypes/struct.Schema.html#method.contains),
which explicitly permits a non-nullable field where the schema declares a
nullable one.
Per review discussion on #24237, this PR contributes just an end-to-end
reproducer as a basis for working out a root-cause fix.
## What changes are included in this PR?
A new test module,
`datafusion/core/tests/sql/aggregates/nested_nullability.rs`. It registers a
`MemTable` whose declared schema marks the nested struct field `b.colA`
nullable while the batches carry a stricter, non-nullable `colA`, then runs
`array_agg(b)` / `array_agg(DISTINCT b)` grouped by another column via
`ctx.sql(...)`, with and without a memory limit that forces the aggregation to
spill.
A pure `.slt` reproducer is not possible: SQL-created tables derive their
schema from their batches, so the schema/batch divergence cannot be constructed
from SQL alone. Registering the `MemTable` from Rust and running the query
through SQL is the closest end-to-end form.
On current `main` the four variants fail three different ways:
| query | memory limit | result on `main` |
|---|---|---|
| `array_agg(b)` | none | panic in `ListArray::new`
(`ArrayAggGroupsAccumulator::evaluate` builds the list item field from the
planner-declared type over actual-typed values) |
| `array_agg(b)` | forces spilling | same panic |
| `array_agg(DISTINCT b)` | none | error: `RowConverter column schema
mismatch` (the `single_distinct_to_groupby` rewrite makes the struct a GROUP BY
key; `GroupValues`' row converter is built from the declared schema but fed the
stricter arrays) |
| `array_agg(DISTINCT b)` | forces spilling | passes |
## Are these changes tested?
The tests are the change. They currently fail (and one panics) on `main`,
demonstrating the bug.
## Are there any user-facing changes?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
