iceberg-kitten opened a new pull request, #2652:
URL: https://github.com/apache/iceberg-go/pull/2652

   ### What changes
   
   `table.WithArrowBufferReuse()` lets callers opt in to reusing released Arrow 
Go allocations within a read. Each read can retain up to 8 MiB of freed buffer 
capacity, and clears its cache at the end of iteration or on early exit. 
Caller-owned batches still outlive the iterator, and callers that supply a 
custom allocator continue to use it directly.
   
   ### Benchmark
   
   A local 4,194,304-row Iceberg scan over the same Parquet files, Go 1.26.3 on 
Windows amd64. The harness verifies row counts and the ID checksum. These are 
medians of ten timings in two process runs per setting, with order reversed in 
the second run. The filtered scan selects two columns and returns one quarter 
of the rows.
   
   | Workload | Workers | Default | Reuse |
   | --- | ---: | ---: | ---: |
   | Full | 1 | 6,735 ms | 3,908 ms |
   | Filtered | 1 | 628 ms | 476 ms |
   | Full | 4 | 466 ms | 487 ms |
   | Filtered | 4 | 81.3 ms | 80.0 ms |
   
   There was significant background load, especially in the single-worker 
measurements. Buffer reuse needs a quieter repeat and an allocation profile 
before making a default change. The first proposal is opt-in, since the 
multithreaded full scan showed a small median slowdown.
   
   ### Correctness
   
   `go test ./table -run 
'^(TestArrowBufferPoolReusesZeroedBuffers|TestReadTasksArrowBufferReuse)$'` 
passed. It covers zeroing, reallocation, the custom allocator path, and 
releasing a batch after the iterator has ended early.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to