fallintoplace opened a new pull request, #2276: URL: https://github.com/apache/iceberg-go/pull/2276
## What Skip partition-key work for files that will not be compacted. ## Why Large scans can contain many ineligible files. Building their keys and empty buckets uses extra time and memory. ## Implementation - Classify files before encoding partition keys. - Store candidate partitions in an indexed slice. - Keep candidate order within each partition. - Cover skipped files in candidate and empty partitions. ## Benchmark End-to-end `PlanCompaction`. Go 1.26.3, Apple M1 Pro, darwin/arm64, `GOMAXPROCS=1`; benchstat median of 10 runs, 500 ms each. Base: `a4fb4d0`. | Files / partitions | Candidates | Before | After | Less time | Allocs/op | | --- | ---: | ---: | ---: | ---: | ---: | | 20k / 256 | 0% | 8.067 ms | 0.329 ms | 95.9% | 60,020 → 0 | | 20k / 256 | 10% | 8.551 ms | 1.727 ms | 79.8% | 62,534 → 8,533 | | 20k / 256 | 50% | 10.563 ms | 6.265 ms | 40.7% | 63,937 → 33,936 | | 20k / 256 | 100% | 14.62 ms | 11.88 ms | 18.7% | 66,430 → 66,432 | | 100k / 1,024 | 10% | 39.931 ms | 8.793 ms | 78.0% | 311,475 → 41,476 | | 100k / 1,024 | 100% | 73.36 ms | 56.06 ms | 23.6% | 326,667 → 326,668 | Reproduce with: ```sh GOMAXPROCS=1 go test ./table/compaction -run '^$' -bench '^BenchmarkPlanCompaction$' -benchmem -benchtime=500ms -count=10 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
