jayzhan211 commented on issue #24704:
URL: https://github.com/apache/datafusion/issues/24704#issuecomment-5672152728
I think the indirection cost can be avoided without mmap:
1. Flat until the first block fills. State stays a plain Vec<T>
until it reaches block capacity B; then that Vec becomes block 0 as-is (no
copy, B is a power of two so doubling lands on it exactly) and growth appends
blocks. Below B nothing changes from today, which covers the low-cardinality
case entirely.
2. B is a row count shared by keys and all accumulators (so block
k of each column forms one batch), around 2^17–2^18 rows: 1–2 MB for a u64
state, roughly one L2. By the time we're in blocked mode the state doesn't fit
in cache anyway, so the loop is bound by cache misses, not by the shift/mask. B
is independent of batch_size: emission slices a block with
min(remaining_in_block, batch_size), so a non-power-of-two batch_size just
yields one short batch per block, never a concat. A block is freed when its
last slice is dropped, and memory over-reporting on slices is bounded to one
block per column instead of the whole table.
3. Single usize index, idx >> k / idx & mask, flat-vs-blocked
resolved once per update_batch, no per-row branch, no division.
mmap needs three platform paths plus a wasm fallback, memory-pool changes
for reserved vs. committed pages, and custom Buffer deallocators for emitted
blocks. I'd keep it in reserve behind the BlocksIndex abstraction and let a
BlockedVec microbench (flat vs. blocked, cache-resident and not) decide whether
there's a gap worth that cost.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
