rluvaton commented on issue #24704:
URL: https://github.com/apache/datafusion/issues/24704#issuecomment-5681642923
> I think this is the only realistic way we will avoid a prerformance
regression
I wanted to avoid this as the implementations up to this point were pretty
complex to support both in a single GroupsAccumulator.
> I don't understand this assertion. I think in previous versions @Rachelint
basically changed the internal implementations of accumulators (to be something
like Vec<...>) and then just used the existing EmitTo::First(n) API. The key
idea was that you knew the size of the n (it is batch_size) so the accumulators
could size their internal blocks to that output size.
I want to keep EmitTo::All but change the signature to return Vec<ArrayRef>
where each array is a block. but this change cannot be done gradually, since
you need restructure your internal data structure around that block size.
> If there was a mismatch (e.g. for versions that can emit early) then the
accumulators did something less efficient (like copying / concating batches
together)
I don't understand this statement
> I don't fully understand what the mmap proposal is
having the mmap allows to treat it as continues slice which simplify
implementations, uses already super optimized implementation for address
translations rather than us reimplementing it. allowing us to split a memory
region into 2 owned arrow buffers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
