fallintoplace opened a new pull request, #1192: URL: https://github.com/apache/arrow-go/pull/1192
### Rationale for this change `buildDataPageV2` creates a new `bytes.Buffer` for every page when joining levels with encoded values. Pages without active dictionary encoding are written synchronously, so their assembly storage does not need to survive the call. Reallocating it for every page adds page-sized allocation churn. ### What changes are included in this PR? - Reuse the column writer scratch buffer for eagerly written DataPageV2 pages. - Keep an owned buffer for pages retained while dictionary encoding is active. - Add a benchmark that writes 64 eager 64 KiB pages with uncompressed and Snappy codecs. Medians from 6 runs on an Apple M1 Pro were: | codec | metric | main | this PR | change | |---|---|---:|---:|---:| | uncompressed | time/op | 1.08 ms | 650 us | -39.6% | | uncompressed | B/op | 4,547,602 | 383,763 | -91.6% | | uncompressed | allocs/op | 461 | 321 | -30.4% | | Snappy | time/op | 9.05 ms | 8.88 ms | -1.9% | | Snappy | B/op | 3,027,460 | 489,403 | -83.8% | | Snappy | allocs/op | 459 | 325 | -29.2% | ### Are these changes tested? - `go test ./parquet/... -count=1` - `go test ./parquet/file -run "^$" -bench "^BenchmarkWriteDataPageV2Eager$" -benchmem -count=6` The Parquet suite includes multi-page DataPageV2 dictionary coverage for the retained-page path. ### Are there any user-facing changes? No. The Parquet output and public API are unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
