This is an automated email from the ASF dual-hosted git repository.
zeroshade pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-go.git
The following commit(s) were added to refs/heads/main by this push:
new 258cb205 perf(arrow/csv): reserve record builders for chunked reads
(#1198)
258cb205 is described below
commit 258cb205d9f9b32c2bab375dce68e5de8d32002c
Author: Minh Vu <[email protected]>
AuthorDate: Fri Aug 14 20:05:42 2026 +0200
perf(arrow/csv): reserve record builders for chunked reads (#1198)
## Summary
* Reserve the configured chunk capacity once before reading each chunked
batch.
* Keep schema inference working by reserving after the first row
initializes the builder.
* Leave the single-row and ReadAll paths unchanged.
## Benchmark
On an Apple M1 Pro, `BenchmarkRead/rows=1000_cols=100_chunks=10`
improved locally from a median of about 36.4 ms to 25.6 ms.
* B/op: about 24.2 MB -> 17.4 MB
* allocs/op: about 163k -> 113k
The gain is chunk-size dependent. Very small batches are already close
to the builders’ minimum capacity.
## Tests
* `PARQUET_TEST_DATA=... go test ./arrow/csv -run ^Test`\n*
`PARQUET_TEST_DATA=... go test -race ./arrow/csv -run ^Test`\n* Full
package test run passed with the local Parquet test data checkout,
excluding the CSV example that needs a separate Arrow CSV fixture.
---
arrow/csv/reader.go | 3 +++
1 file changed, 3 insertions(+)
diff --git a/arrow/csv/reader.go b/arrow/csv/reader.go
index f2a1f0b2..68d50bc9 100644
--- a/arrow/csv/reader.go
+++ b/arrow/csv/reader.go
@@ -341,6 +341,9 @@ func (r *Reader) nextn() bool {
}
r.validate(recs)
+ if n == 0 && r.err == nil {
+ r.bld.Reserve(r.chunk)
+ }
r.read(recs)
n++
}
