iceberg-kitten opened a new pull request, #1625:
URL: https://github.com/apache/arrow-go/pull/1625

   ### What changes
   
   The dense Parquet byte-array reader knows the decoded value lengths before 
appending them to an Arrow binary or string builder. For payloads up to 8 MiB, 
reserve the offsets and data before appending. Larger payloads use incremental 
appends. This keeps Arrow output types and null semantics intact.
   
   ### Benchmark
   
   `BenchmarkReadDenseByteArrays` reads 65,536 non-null strings in 8,192-row 
batches. Go 1.26.3 on Windows amd64, one pinned performance core, medians of 
four interleaved runs against `4f4ac82`.
   
   | Parquet encoding | Base allocated bytes/op | This branch | Median elapsed 
time |
   | --- | ---: | ---: | ---: |
   | Plain | 6.62 MB | 5.43 MB | 15.75 ms → 17.29 ms |
   | Dictionary | 6.62 MB | 5.43 MB | 20.87 ms → 15.80 ms |
   
   Both encodings allocated about 18% fewer bytes and 20% fewer objects. 
Timings varied substantially under concurrent builds. This draft is for the 
allocation result pending a quieter latency run.
   
   ### Correctness
   
   `go test ./parquet/pqarrow -run '^TestReadDenseByteArrays$'` passed. The 
round trip exercises plain and dictionary encodings, empty strings, and the 
large-value fallback using a checked allocator.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to