jayzhan211 commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-5557732347
> Why do you need `EmitTo::Last` for this? Can't you simply keep producing
the large RecordBatch as it's being done now (via EmitTo::All or
EmitTo::First(n) in case of partial aggregation early emit, where n is very
close to the total number of groups) and simply create separate allocations for
the individual Record Batches instead of slicing them?
Here's a diagram — hope it helps!
```
Memory held by the aggregation while it drains N groups (batch_size = B)
today: EmitTo::All + slice copy-out per batch
EmitTo::Last(n)
─────────────────────────── ──────────────────────
───────────────────────
┌───────────────────────┐ ┌───────────────────────┐
┌───────────────────────┐
│ one giant batch │ │ one giant batch │ │
Vec / blocks │
│ [ b0 | b1 | b2 | b3 ] │ │ [ b0 | b1 | b2 | b3 ] │ │
[ b0 | b1 | b2 | b3 ] │
└───────────────────────┘ └───────────────────────┘
└───────────────────────┘
│ slices share the buffer │ copy b0 → own alloc
truncate / pop b3 ──▶ out
▼ ▼
┌──────────────────┐
b0 out (giant still live) b0 out (giant still live)
│ [ b0 | b1 | b2 ] │ b3 freed
b1 out (giant still live) b1 out (giant still live)
└──────────────────┘
b2 out (giant still live) b2 out (giant still live)
[ b0 | b1 ] b2 freed
b3 out → giant freed b3 out → giant freed
[ b0 ] b1 freed
[] b0 freed
held ▲ held ▲
held ▲
│████████████████ ███ │████████████████
███ │██
│████████████████ ███ │████████████████
███ │████
│████████████████ │████████████████
│██████
└──────────────── time ▶ └──────────────── time ▶
└──────────────── time ▶
all of it, until the end same, +transient copies
shrinks one batch per poll
First(n): shift the rest down + renumber every remaining index ──▶
O(remaining) per call
Last(n): drop the tail ──▶
nothing moves, nothing renumbers
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
