fallintoplace opened a new pull request, #1765:
URL: https://github.com/apache/iceberg-go/pull/1765
## What changed
- Build partition extraction descriptors once from the first record batch.
- Share the immutable plan across fanout workers.
- Reuse the same plan in clustered writes.
- Rebuild for a batch only if its Arrow schema actually changes.
- Keep the existing literal conversion and transform behavior unchanged.
## Why
Partition routing rebuilt the partition type, field lookup map, source
names, source types, and Arrow column indexes for every batch. This fixed setup
cost is noticeable for streams with many small batches.
The plan is built from the first real batch because partitioned
equality-delete input may include routing columns that are omitted from the
output file schema.
## Benchmark
Apple M1 Pro, median of 3 runs:
go test ./table -run ^$ -bench ^BenchmarkPartitionExtraction$ -benchmem
-benchtime=1s -count=3 -cpu=1
| Rows per batch | main | this PR | main allocations | this PR allocations |
| ---: | ---: | ---: | ---: | ---: |
| 0 | 1695 ns | 197 ns | 1432 B, 10 allocs | 128 B, 2 allocs |
| 1 | 3926 ns | 2412 ns | 3928 B, 28 allocs | 2624 B, 20 allocs |
| 16 | 20558 ns | 18070 ns | 21456 B, 150 allocs | 20152 B, 142 allocs |
| 1024 | 184282 ns | 156782 ns | 21456 B, 150 allocs | 20152 B, 142 allocs |
## Tests
- go test ./...
- go test -race ./table with the fanout, clustered, equality-delete, and
shredded-variant writer tests
- go vet ./...
- git diff --check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
