iceberg-kitten opened a new pull request, #2652: URL: https://github.com/apache/iceberg-go/pull/2652
### What changes `table.WithArrowBufferReuse()` lets callers opt in to reusing released Arrow Go allocations within a read. Each read can retain up to 8 MiB of freed buffer capacity, and clears its cache at the end of iteration or on early exit. Caller-owned batches still outlive the iterator, and callers that supply a custom allocator continue to use it directly. ### Benchmark A local 4,194,304-row Iceberg scan over the same Parquet files, Go 1.26.3 on Windows amd64. The harness verifies row counts and the ID checksum. These are medians of ten timings in two process runs per setting, with order reversed in the second run. The filtered scan selects two columns and returns one quarter of the rows. | Workload | Workers | Default | Reuse | | --- | ---: | ---: | ---: | | Full | 1 | 6,735 ms | 3,908 ms | | Filtered | 1 | 628 ms | 476 ms | | Full | 4 | 466 ms | 487 ms | | Filtered | 4 | 81.3 ms | 80.0 ms | There was significant background load, especially in the single-worker measurements. Buffer reuse needs a quieter repeat and an allocation profile before making a default change. The first proposal is opt-in, since the multithreaded full scan showed a small median slowdown. ### Correctness `go test ./table -run '^(TestArrowBufferPoolReusesZeroedBuffers|TestReadTasksArrowBufferReuse)$'` passed. It covers zeroing, reallocation, the custom allocator path, and releasing a batch after the iterator has ended early. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
