Rich-T-kid commented on issue #7764: URL: https://github.com/apache/arrow-rs/issues/7764#issuecomment-5384281798
The optimization described here (pre-allocating exactly the required buffer space and writing directly) was attempted in #10799. The implementation defers all copies to `finish()`, storing (offset, len, Arc) tuples per copy_rows call, then scans to compute total byte and row counts before a single allocation and copy pass. The problem is concat already does this for byte arrays. binary_capacity in https://github.com/apache/arrow-rs/blob/cd7c6b83abd6605a83014b3d043930a592542510/arrow-select/src/concat.rs#L48 scans value_offsets to compute exact byte counts, and concat_bytes https://github.com/apache/arrow-rs/blob/cd7c6b83abd6605a83014b3d043930a592542510/arrow-select/src/concat.rs#L355 uses those to pre-allocate before writing. Benchmarks from #10799 showed results within noise (+/-5) of GenericInProgressArray, because we're doing logically the same work: scan offsets -> pre-allocate -> copy. Unlike ByteView format where strings <=12 bytes are stored inline in the view itself (no value buffer touched), StringArray/BinaryArray always requires copying through the value buffer. The 30-50% win only applies to the view format." I'm going to close #10799 & open #10797 for review since it may to useful to anyone who attempts this issue again 🚀 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
