alamb commented on issue #8850: URL: https://github.com/apache/arrow-rs/issues/8850#issuecomment-3543344786
> If you don't know the sizes up front it might be more performant to buffer up the record batch and then pass slices to the concat_batches kernel. Otherwise you will end up bump allocating, which is not normally very performant. The "GenericInProgressArray" here already basically already implements buffer + concat: https://github.com/apache/arrow-rs/blob/main/arrow-select/src/coalesce/generic.rs Maybe we could use that somehow (to avoid having to know the total size upfront) 🤔 Or maybe we could add a setting on BatchCoalescer -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
