fallintoplace opened a new pull request, #1768: URL: https://github.com/apache/iceberg-go/pull/1768
Depends on #1765. This is stacked on top of it, so only the second commit is new here. ## What changed - Bind built-in partition transform functions when the extraction plan is created. - Feed native Iceberg values directly into identity, bucket, truncate, and time transforms. - Keep the generic literal path for custom or unsupported transforms. - Keep unknown and void transforms producing null partition values. ## Why The row loop currently converts each Arrow value into a literal and calls `Transform.Apply`. Some transforms do more setup there. For example, `TruncateTransform.Apply` rebuilds its transformer for every value. The extraction plan has the source type already, so it can bind the transform once and reuse the function for every batch. Java's `StructTransform` uses the same bind-once shape. ## Benchmark Apple M1 Pro, 65,536 rows, median of 3 runs. The baseline is #1765. ``` go test ./table -run '^$' -bench '^BenchmarkPartitionTransforms$' -benchmem -benchtime=1s -count=3 -cpu=1 ``` | Transform | #1765 | this PR | | --- | ---: | ---: | | identity int64 | 2,648,326 ns/op | 2,531,346 ns/op | | bucket string | 5,862,667 ns/op | 5,710,295 ns/op | | truncate string | 15,767,652 ns/op, 7,271,656 B/op, 394,124 allocs/op | 7,205,658 ns/op, 3,077,352 B/op, 131,980 allocs/op | | day timestamp | 3,898,547 ns/op | 3,468,050 ns/op | | hour timestamp_ns | 3,812,458 ns/op | 3,621,065 ns/op | The largest change is `truncate[string]`, which is about 2.2x faster with 67% fewer allocations. ## Tests - `go test ./...` - `go test -race ./table -run '^TestFanoutWriter$' -count=1` - `go vet ./...` - `git diff --check` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
