fallintoplace opened a new pull request, #1193: URL: https://github.com/apache/arrow-go/pull/1193
### Rationale for this change `PlainByteArrayEncoder.Put` currently calls `PutByteArray` for every value. Each call checks sink capacity, writes the 4-byte length, then writes the payload. Large batches repeat that sink work for every value. ### What changes are included in this PR? - Calculate the encoded batch size in one pass. - Reserve the sink capacity once. - Write lengths and payloads directly into the reserved region. - Add benchmarks for `Put` and all-valid `PutSpaced` with 1,048,576 values at three widths. The single-value `PutByteArray` path is unchanged. `PutSpaced` benefits through its existing valid-run batching. Medians from 6 runs on an Apple M1 Pro were: | method | value width | main | this PR | change | |---|---:|---:|---:|---:| | Put | 4 bytes | 10.25 ms | 3.85 ms | -62.4% | | Put | 16 bytes | 11.00 ms | 4.38 ms | -60.2% | | Put | 64 bytes | 13.77 ms | 7.00 ms | -49.2% | | PutSpaced | 4 bytes | 10.33 ms | 3.95 ms | -61.8% | | PutSpaced | 16 bytes | 11.13 ms | 4.44 ms | -60.1% | | PutSpaced | 64 bytes | 13.54 ms | 7.45 ms | -45.0% | Allocation counts are unchanged. ### Are these changes tested? - `go test ./parquet/... -count=1` - `go test ./parquet/internal/encoding -run "^$" -bench "^BenchmarkEncodePlainByteArray$" -benchmem -count=6` The existing encoding suite covers plain BYTE_ARRAY round trips and spaced validity patterns. ### Are there any user-facing changes? No. The encoded Parquet bytes and public API are unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
