fallintoplace opened a new pull request, #1175: URL: https://github.com/apache/arrow-go/pull/1175
### Rationale for this change Plain Boolean spaced encoding currently counts valid bits, allocates a compacted bitmap, copies valid runs into it, and then copies that bitmap into the encoder buffer. Short unaligned runs also create bitmap readers and writers for every copy. This is expensive for common nullable patterns. ### What changes are included in this PR? - Append valid bitmap runs directly to the existing encoder buffer. - Accumulate the valid count while encoding instead of scanning validity first. - Write short runs directly to avoid unaligned bitmap-copy allocations. - Keep bulk bitmap copies for longer runs. - Share bitmap writer initialization between the Boolean input paths. - Add benchmarks for 1K, 64K, and 1M values across several null patterns. - Add tests for bitmap offsets, validity offsets, encoder-buffer boundaries, and repeated calls. Apple M1 Pro results for 65,536 values with `-cpu=1`: | Validity pattern | Before | After | Speedup | | --- | ---: | ---: | ---: | | All valid | 6.77 us | 3.00 us | 2.26x | | 1% null | 224 us | 182 us | 1.23x | | 10% null | 912 us | 307 us | 2.97x | | Alternating 50% null | 3.86 ms | 394 us | 9.79x | | 90% null | 1.16 ms | 95.8 us | 12.1x | | Clustered nulls | 7.53 us | 3.65 us | 2.06x | For the short-run patterns: - 10% null: 847 KB and 13,109 allocations to 23 B and 0 allocations. - 50% null: 4.20 MB and 65,542 allocations to 14 B and 0 allocations. - 90% null: 735 KB and 11,469 allocations to 0 B and 0 allocations. ### Are these changes tested? Yes. - `go test ./parquet/...` - `go test -race ./parquet/internal/encoding` - `go vet -composites=false ./parquet/internal/encoding` - Cross-compiled the encoding tests for linux/amd64 and linux/s390x. ### Are there any user-facing changes? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
