fallintoplace opened a new pull request, #1328: URL: https://github.com/apache/arrow-go/pull/1328
### What - Encode Parquet dictionary indices in RLE batches. ### Why - `WriteIndices` currently calls `Put` for every index, including long repeated runs. ### Implementation - Add `PutBatchIndices` for `int32` indices. - Accumulate repeated runs in bulk and fill complete literal groups directly. - Keep scalar handling for partial literal groups. - Preserve encoded bytes and dictionary state on write errors. ### Benchmark Apple M1 Pro, Go 1.26.3, `GOMAXPROCS=1`. `WriteIndices` on 65,536 indices. Medians of six 500 ms samples per version, alternating before/after order. | Input | Before | After | Time change | | --- | ---: | ---: | ---: | | Constant | 187.7 µs | 26.0 µs | -86.1% | | Runs of 8 | 388.9 µs | 200.9 µs | -48.3% | | Runs of 32 | 219.4 µs | 67.0 µs | -69.5% | | Runs of 256 | 197.6 µs | 41.9 µs | -78.8% | | Random, 16 entries | 580.7 µs | 306.3 µs | -47.3% | | Random, 256 entries | 452.8 µs | 302.8 µs | -33.1% | | Alternating | 400.7 µs | 270.6 µs | -32.5% | Allocations stay at 232 B/op and 4 allocs/op. The benchmark also covers 64-index batches. ### Tests - `go test ./parquet/...` - `go test -race ./parquet/internal/utils ./parquet/internal/encoding` - `go test -tags noasm ./parquet/internal/utils ./parquet/internal/encoding` - `go vet -composites=false ./parquet/internal/utils ./parquet/internal/encoding` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
