fallintoplace opened a new pull request, #1901: URL: https://github.com/apache/iceberg-go/pull/1901
## What changed - Skip task filter extraction when `Residual` is nil. - Keep binding and extraction for non-nil task residuals. - Add a mixed-task correctness test. - Add a benchmark for 1K, 10K, and 100K tasks with a complex filter. ## Why `scanInvariants` already extracts the bound scan filter once. Normal tasks have no residual, so extracting that same filter again for every task only repeats work. ## Benchmark Run on an Apple M1 Pro with: `go test ./table -run '^$' -bench 'BenchmarkArrowScanAddTaskProjectedFieldIDs/tasks=(1000|10000|100000)/residual=nil$' -benchmem -count=5` These are medians over five runs. Before is `upstream/main` and after is this change. | Tasks | Before | After | | --- | --- | --- | | 1K | 907.823 µs/op, 1.20 MB/op, 13,000 allocs/op | 2.995 µs/op, 0 B/op, 0 allocs/op | | 10K | 8.932 ms/op, 12.00 MB/op, 130,000 allocs/op | 31.966 µs/op, 0 B/op, 0 allocs/op | | 100K | 82.351 ms/op, 120.00 MB/op, 1,300,000 allocs/op | 342.928 µs/op, 0 B/op, 0 allocs/op | ## Testing - `go test ./table -count=1` - `go test ./table -race -count=1` - `go vet ./table` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
