fallintoplace opened a new pull request, #1963: URL: https://github.com/apache/iceberg-go/pull/1963
# Summary - **Load equality deletes per scan task.** - Keep a shared, read-once cache for delete files and merged file combinations. - Start scan workers when the iterator is consumed, so an unread iterator does no equality-delete I/O. - Keep equality-field grouping and delete-set reuse. The old path read every unique equality-delete file before returning the iterator. This change only reads files needed by tasks that are actually processed. # Benchmark Setup: 10,000 scan tasks, 1,000 equality-delete files, 10 keys per file, 16 workers. Three runs with `-benchtime=1s` on an Apple M1 Pro. | case | time/op | bytes/op | allocs/op | opens/op | | --- | ---: | ---: | ---: | ---: | | eager/all_tasks | 12.8 to 13.7 ms | 39.6 MB | 299k | 1,000 | | lazy/unread | 0.45 to 0.49 ms | 285 KB | 11,023 | 0 | | lazy/first_task | 0.50 to 0.54 ms | 326 KB | 11,298 | 1 | | lazy/ten_tasks | 0.65 to 0.75 ms | 666 KB | 13,713 | 10 | | lazy/full_scan | 13.1 to 14.6 ms | 39.2 MB | 287k | 1,000 | - **Unread iterator:** no equality-delete opens. - **Partial scans:** I/O follows the tasks reached. - **Full scan:** comparable runtime with fewer allocations. # Tests - `go test ./table -count=1` - `go test -race ./table -count=1` - `go vet ./table` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
