alamb commented on issue #24704: URL: https://github.com/apache/datafusion/issues/24704#issuecomment-5686786185
> I want to keep EmitTo::All but change the signature to return Vec where each array is a block. but this change cannot be done gradually, since you need restructure your internal data structure around that block size. Right, I was suggesting we don't change the signature (still only return a single RecordBatch) but change the caller s to drain data via `EmitTo` batches. Maybe we could defer index updates, etc > > If there was a mismatch (e.g. for versions that can emit early) then the accumulators did something less efficient (like copying / concating batches together) > > I don't understand this statement I was trying to imply that if batch_size was 8192, but for some reason EmitTo::First(10) was called, then internally the accumulator would have to shift state around -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
