mohitgurav20 opened a new pull request, #27196:
URL: https://github.com/apache/datafusion/pull/27196
## Which issue does this PR close?
- Closes #26099.
- Part of #23393.
## Rationale
Partial group clustering keeps the current run's key in a
`Vec<ScalarValue>`. The memory accounting only summed the live elements, so
spare capacity in the vector was never reported.
This happens in practice: `get_row_at_idx` allocates room for four scalars
and fills one, so three scalar slots stay unreported. Since these numbers
drive spill and OOM decisions, under-reporting can delay a spill and turn a
manageable query into an OOM.
## What changed
- `State::heap_size` now uses `DFHeapSize` for the sort key. It charges the
vector's full capacity and the heap payload of each initialized scalar
once.
- `GroupClusteringPartial::heap_size` passes a `DFHeapSizeCtx`, matching the
grouped `COUNT` accumulator.
- The inline `Vec` descriptor is still owned by `GroupClustering::size()`,
so nothing is double-counted.
- Updated the existing retained-allocation test and added a focused test
with spare capacity and a heap-owning scalar.
Accounting runs once per batch in `update_memory_reservation`, not per row,
so there is no throughput cost.
## Testing
The new test uses an in-progress key with capacity 4 and one live element,
plus a `Utf8` payload. It checks the exact retained bytes and that live-only
accounting is lower, so a regression fails. The existing partial-clustering
test now derives the expected key size from `ScalarValue::size_of_vec`,
covering the real extraction path.
## User-facing changes
None. Only internal memory accounting changes; query results and public APIs
are unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
