(arrow-rs) branch main updated: perf: Write compressed directly to buffer (#10833)

[alamb]

This is an automated email from the ASF dual-hosted git repository.

alamb pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-rs.git


The following commit(s) were added to refs/heads/main by this push:
     new 0a8fdd505b perf: Write compressed directly to buffer (#10833)
0a8fdd505b is described below

commit 0a8fdd505bbf1b551a7c0654f879b4bde7e41f18
Author: Emily Matheys <55631053+emilym...@users.noreply.github.com>
AuthorDate: Wed Aug 26 16:10:15 2026 +0300

    perf: Write compressed directly to buffer (#10833)
    
    Currently we have to allocate twice when compressing using zstd - once
    to a temporary vec, which is the output from the compress(input) call,
    and then extend our output buffer with extend_from_slice which causes
    another realloc(especially if writing a large buffer)
    instead we can use the zstd compress_to_buffer function directly, which
    is what the compress() call does anyway.
    
    ---------
    
    Co-authored-by: Daniël Heres <danielhe...@gmail.com>
---
 arrow-ipc/src/compression.rs | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/arrow-ipc/src/compression.rs b/arrow-ipc/src/compression.rs
index 176ad726a6..c40ee56055 100644
--- a/arrow-ipc/src/compression.rs
+++ b/arrow-ipc/src/compression.rs
@@ -328,8 +328,15 @@ fn compress_zstd(
     context: &mut IpcWriteContext,
     level: i32,
 ) -> Result<(), ArrowError> {
-    let result = context.zstd_compressor(level).compress(input)?;
-    output.extend_from_slice(&result);
+    let start = output.len();
+    output.reserve(zstd::zstd_safe::compress_bound(input.len()));
+
+    let mut cursor = std::io::Cursor::new(output);
+    cursor.set_position(start as u64);
+    context
+        .zstd_compressor(level)
+        .compress_to_buffer(input, &mut cursor)?;
+
     Ok(())
 }