fallintoplace opened a new pull request, #1904: URL: https://github.com/apache/iceberg-go/pull/1904
## What changed - Cache parsed manifest-list descriptors for one conflict-validation context. - Reuse completed manifest reads by path across validators and snapshots. - Keep the first scan streaming and do not retain early exits. - Add shared-read and early-exit correctness coverage. - Add a benchmark that reports backend opens and bytes fetched. ## Why A commit can run more than one conflict validator against the same `conflictContext`. Each pass used to reopen the same manifest list and manifest file, which added extra object-store reads. The cache only lives for one validation attempt. Manifest entry decoding still runs with each descriptor, so inheritance behavior stays unchanged. ## Benchmark Run on an Apple M1 Pro with: `go test ./table -run '^$' -bench '^BenchmarkConflictValidationSharedManifestReads/(entries=(1000|10000))/(validators=(2|8))$' -benchmem -benchtime=1s -count=5` These are medians over five runs. Before is `upstream/main` at `c9813324`. The backend metrics are per commit attempt. Local wall time is mostly Avro decoding and is more noisy. | Workload | Before | After | | --- | --- | --- | | 1K entries / 2 validators | 4 opens, 16,160 B, 21,302 allocs | 2 opens, 8,080 B, 19,954 allocs | | 1K entries / 8 validators | 16 opens, 64,640 B, 85,214 allocs | 2 opens, 8,080 B, 75,718 allocs | | 10K entries / 2 validators | 4 opens, 63,976 B, 148,189 allocs | 2 opens, 31,988 B, 146,843 allocs | | 10K entries / 8 validators | 16 opens, 255,904 B, 592,759 allocs | 2 opens, 31,988 B, 583,261 allocs | ## Testing - `go test ./table -count=1` - `go test ./table -race -count=1` - `go vet ./table` - `go test ./... -run '^$'` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
