iceberg-kitten opened a new pull request, #1625: URL: https://github.com/apache/arrow-go/pull/1625
### What changes The dense Parquet byte-array reader knows the decoded value lengths before appending them to an Arrow binary or string builder. For payloads up to 8 MiB, reserve the offsets and data before appending. Larger payloads use incremental appends. This keeps Arrow output types and null semantics intact. ### Benchmark `BenchmarkReadDenseByteArrays` reads 65,536 non-null strings in 8,192-row batches. Go 1.26.3 on Windows amd64, one pinned performance core, medians of four interleaved runs against `4f4ac82`. | Parquet encoding | Base allocated bytes/op | This branch | Median elapsed time | | --- | ---: | ---: | ---: | | Plain | 6.62 MB | 5.43 MB | 15.75 ms → 17.29 ms | | Dictionary | 6.62 MB | 5.43 MB | 20.87 ms → 15.80 ms | Both encodings allocated about 18% fewer bytes and 20% fewer objects. Timings varied substantially under concurrent builds. This draft is for the allocation result pending a quieter latency run. ### Correctness `go test ./parquet/pqarrow -run '^TestReadDenseByteArrays$'` passed. The round trip exercises plain and dictionary encodings, empty strings, and the large-value fallback using a checked allocator. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
