fallintoplace opened a new pull request, #1769: URL: https://github.com/apache/iceberg-go/pull/1769
## What changed - Keep the existing maximum starting capacity of 128 row indexes. - Reduce the starting capacity as more partitions are discovered in a batch. - Use `batch rows / discovered partitions` as the estimate, with a minimum of one. - Keep grouping and row ordering unchanged. ## Why Every new partition currently reserves space for 128 row indexes. This works well for low-cardinality batches, but a batch with one partition per row reserves about 1 KiB for every single row. The adaptive estimate keeps the current behavior for a few large partitions. It gradually moves toward one row of capacity for high-cardinality batches. ## Benchmark Apple M1 Pro, 32,768 rows, median of 5 runs: ``` go test ./table -run '^$' -bench '^BenchmarkPartitionRowCapacity$' -benchmem -benchtime=1s -count=5 -cpu=1 ``` | Distribution | main | this PR | Change | | --- | ---: | ---: | ---: | | 1 partition | 1,160,120 B/op | 1,160,120 B/op | unchanged | | 16 partitions | 951,784 B/op | 951,784 B/op | unchanged | | 1,024 partitions | 1,759,064 B/op | 1,348,728 B/op | -23% | | 1 partition per row | 50,331,068 B/op | 18,241,259 B/op | -64% | | 90% in one partition | 5,994,673 B/op | 3,580,624 B/op | -40% | Allocation counts are unchanged in every case. The unique-partition case was also about 14% faster in the repeated comparison. ## Tests - `go test ./...` - `go test -race ./table -run '^TestFanoutWriter$' -count=1` - `go vet ./...` - `git diff --check` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
