fallintoplace opened a new pull request, #2379: URL: https://github.com/apache/iceberg-go/pull/2379
## What - Bound the temporary paths held while reading orphan-cleanup manifests. ## Why - Large manifests currently buffer every path before adding it to the referenced set. ## Implementation - Flush paths to the shared set in batches of 512. - Grow the buffer as needed for small manifests. - Keep data-file and deletion-vector references. ## Benchmark `getReferencedFiles`, one in-memory manifest, Apple M1 Pro, Go 1.26.3, median of 5 runs, `GOMAXPROCS=1`: | Entries | Before | After | Allocated bytes before → after | | --- | ---: | ---: | ---: | | 1 | 0.620 ms | 0.582 ms | 480 KB → 480 KB | | 16,384 | 20.77 ms | 18.39 ms | 14.84 MB → 13.12 MB | `GOMAXPROCS=1 go test ./table/maintenance -run '^$' -bench '^BenchmarkGetReferencedFilesManifestEntries$' -benchmem -benchtime=200ms -count=5 -cpu=1` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
