shehabgamin opened a new issue, #26744:
URL: https://github.com/apache/datafusion/issues/26744
### Describe the bug
In DataFusion 55.2.0, `DISTINCT ON` queries carrying nested non-key columns
can fail with a memory reservation error during aggregate spill/replay.
The `FIRST_VALUE` accumulator's merge path retains backing buffers from the
larger input array instead of compacting the selected scalar. This inflates
per-group memory accounting.
### To Reproduce
The failure was reproduced with DataFusion 55.2.0 using this workload:
1. Register a `MemTable` containing two partitions, each with one 4,096-row
batch.
2. Use an `Int64` key with 8,192 distinct values and a payload of type
`List<Struct<type: Utf8, value: Utf8>>`. Each list contains one struct with
`type = "message"` and `value = "v".repeat(224)`.
3. Configure a 16 MiB `FairSpillPool`, two target partitions, and disk
spilling.
4. Execute and collect:
```rust
let batches = ctx
.table("t")
.await?
.distinct_on(
vec![col("k")],
vec![col("k"), col("contents")],
None,
)?
.collect()
.await?;
```
The unpatched implementation fails after two aggregate spills when
`FinalHashAggregateStream` requests approximately 4.3 GB from the memory pool.
### Expected behavior
The query should complete using aggregate spilling and return the correct
rows. Merging a selected nested value should compact its backing buffers,
consistent with the accumulator's update path.
### Additional context
_No response_
### Willingness to contribute
I plan to contribute a fix for this bug independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
