fallintoplace opened a new pull request, #1287: URL: https://github.com/apache/arrow-go/pull/1287
### Rationale for this change `GetTakeIndices` currently grows a buffer once per selected run. Fragmented filters such as alternating and short random runs create many allocations and are much slower than dense filters. ### What changes are included in this PR? - Add an ARM64 NEON path for large, byte-aligned, non-null `uint32` filters. - Compact 4-bit mask lookups into exact-size take-index output. - Keep `uint16`, nullable, unaligned, small, dense, long-run, and `EmitNulls` paths unchanged. - Add coverage for all 4-bit masks, offsets, tails, padding bits, null behavior, and width boundaries. - Add GetTakeIndices and large FilterRecordBatch benchmarks. **Benchmark hardware:** Apple M1 Pro, macOS arm64 **Commands:** ```text go test ./arrow/compute/internal/kernels -run '^$' -bench='^BenchmarkGetTakeIndices/(all-selected|all-clear|long-runs|alternating|short-runs|random)/' -benchmem -benchtime=3x -count=3 go test ./arrow/compute -run '^$' -bench='^BenchmarkFilterRecordBatchGetTakeIndices/(rows=65536|rows=1048576)$' -benchmem -benchtime=1x -count=3 ``` Median of the three reported benchmark samples. Values are ns/op per call. | Workload | Selection pattern | Existing | NEON path | Speedup | | --- | --- | ---: | ---: | ---: | | GetTakeIndices 64K | all-selected | 41.8 us | 41.2 us | 1.0x | | GetTakeIndices 64K | all-clear | 2.54 us | 3.03 us | 0.84x | | GetTakeIndices 64K | long-runs | 826 us | 923 us | 0.89x | | GetTakeIndices 64K | alternating | 48.1 ms | 21.9 us | 2,198x | | GetTakeIndices 64K | short-runs | 4.87 ms | 22.6 us | 215x | | GetTakeIndices 64K | random | 73.5 ms | 90.0 us | 817x | | GetTakeIndices 1M | all-selected | 1.80 ms | 0.80 ms | 2.3x* | | GetTakeIndices 1M | all-clear | 37.8 us | 39.0 us | 0.97x | | GetTakeIndices 1M | long-runs | 705 ms | 176 ms | 4.0x* | | GetTakeIndices 1M | alternating | 4.31 s | 515 us | 8,369x | | GetTakeIndices 1M | short-runs | 201 ms | 230 us | 873x | | GetTakeIndices 1M | random | 3.56 s | 1.69 ms | 2,111x | `*` These patterns use the existing fallback. The measured difference is run-to-run allocator and GC noise, not a new fast path. Downstream serial `FilterRecordBatch` with one Int64 column and an alternating filter measured **17.7 ms to 152 us (116x)** at 64K rows and **2.06 s to 831 us (2,477x)** at 1M rows. This benchmark includes both take-index generation and the downstream Take call. ### Are these changes tested? - `go test ./...` with the Arrow and Parquet test-data submodules initialized and `PARQUET_TEST_DATA` / `ARROW_TEST_DATA` set. - `go test ./arrow/compute/...` - `go test -tags noasm ./arrow/compute/internal/kernels -run 'TestGetTakeIndices(BatchedRanges|Uint32Coverage)$' -count=1` - `GOOS=linux GOARCH=amd64 go test -c ./arrow/compute/internal/kernels -o /dev/null` ### Are there any user-facing changes? No API changes. The optimization is ARM64-only and only applies to the narrow `uint32`, non-null, `DropNulls` case. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
