andygrove opened a new issue, #27240:
URL: https://github.com/apache/datafusion/issues/27240
### Describe the bug
An aggregate whose state holds a large dictionary can still fail with
`Failed to allocate additional ...` after #25853 (which fixes #25851), when the
dictionary values repeat within a group but not across groups. #25853 sizes a
dictionary by the values its keys reference only when fewer than two keys
reference each value on average; otherwise it counts the keys only and leaves
the dictionary shared, as on main. A dictionary whose values repeat a few times
within a group, but whose groups share no values, takes the shared path, and
spill and replay batches still carry the whole dictionary.
### To Reproduce
Measured with `datafusion-cli` (`release-nonlto`) at the head of #25853
(29c666126):
```sql
SET datafusion.execution.target_partitions = 1;
SELECT count(*), sum(n) FROM (
SELECT (v / 10) % 32 AS k,
cardinality(array_agg(arrow_cast(concat(CAST(v / 10 AS VARCHAR),
repeat('x', 100)), 'Dictionary(Int32, Utf8)'))) AS n
FROM generate_series(1, 5000000) AS t(v)
GROUP BY (v / 10) % 32);
```
`datafusion-cli -m 24M` fails with `Failed to allocate additional 24.7 MB
for SingleHashAggregateStream[0] with 0.0 B already allocated for this
reservation`. It passes at `-m 64M`.
Changing only how often each value repeats in a group (`v / 10` in both
places):
| repeats per value | `-m 24M` |
| --- | --- |
| 2 | passes |
| 10 | fails |
| 50 | passes |
With 1000-byte values and `target_partitions = 2`, the final aggregate fails
at `-m 24M` (`Failed to allocate additional 16.8 MB for
FinalHashAggregateStream[0]`). With 1000-byte values and `target_partitions =
1` over 2M rows, it fails at both 24M and 64M.
### Expected behavior
The query completes under a memory limit large enough for one group, as the
other repeat counts do.
### Additional context
Compacting the dictionary of each spilled chunk would be cheap here, but
whether it pays off depends on whether the groups share their values. If they
do, each chunk writes its own copy and the spill file grows, which is why
#25853 doesn't compact every dictionary (`array_agg` over 100 distinct 200-byte
values spilled 371 MB, against 14.7 MB without compaction). Telling the two
cases apart needs the number of distinct values referenced per row, which
`SpilledRowSizes` doesn't track for `List<Dictionary>`.
Related: #25851, #25853, #26711.
### Willingness to contribute
I would like to contribute a fix for this bug, but request guidance from the
DataFusion community
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
