fallintoplace opened a new pull request, #1754: URL: https://github.com/apache/iceberg-go/pull/1754
## Summary Equality delete files are already deduplicated and read once, but their decoded key maps are copied into a fresh union for every scan task. When many tasks share the same delete files, setup time and memory therefore grow with the number of tasks rather than the number of distinct delete-file combinations. This change reuses the decoded key sets during per-task assembly: * a group backed by one delete file points directly at that file's key set * multi-file groups are canonicalized by delete-file ID and identical combinations share one merged set * duplicate references to the same delete file are removed before building a union * delete files with different equality field IDs remain separate, as before The shared sets are immutable during record filtering, so the scan hot path still performs one lookup per equality-field group. It does not add a chain of underlying maps or change row filtering behavior. This is independent of #1752. That PR speeds up deciding which equality delete files apply during planning; this change avoids repeatedly materializing their decoded key unions before reading rows. ## Benchmarks Representative medians from five runs with three iterations each on an Apple M1 Pro: ```text workload copy per task shared speedup memory before memory after 1 file, 1 combination 68.27 ms 0.31 ms 219x 109.3 MB 0.22 MB 4 files, 1 combination 71.98 ms 0.87 ms 83x 109.4 MB 0.49 MB 4 files, 100 combinations 71.24 ms 8.57 ms 8.3x 109.4 MB 11.30 MB 2 files, 1,000 unique combos 10.97 ms 10.78 ms 1.0x 13.8 MB 13.97 MB ``` Each workload has 1,000 tasks. The shared cases model global or partition-level deletes used by multiple data files. The fully unique case is included as a control and remains effectively neutral. Command: ```sh go test ./table -run '^$' -bench '^BenchmarkEqualityDeleteSetAssembly$' -benchmem -benchtime=3x -count=5 ``` ## Testing * `go test ./...` * `go test -race ./table/...` * `go vet ./...` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
