fallintoplace opened a new pull request, #1935: URL: https://github.com/apache/iceberg-go/pull/1935
## Summary - **Use manifest-list row counts to stop local planning after enough rows are covered.** - Keep the existing full planner for row filters, delete manifests, unknown counts, overflow, and limits that need the full list. - Keep exact row-limit enforcement in the record reader. - Add boundary tests, a manifest-open regression test, and a planning benchmark. ⚡ ## Why With no row filter and no delete manifests, all live rows in the selected data manifests are returned. The manifest list already has the live row counts, so local planning can stop opening manifests once the requested minimum is covered. ## Benchmark **Machine:** Apple M1 Pro, arm64 **Setup:** in-memory v2 table, one data file and one row per manifest, max concurrency 1 **Command:** `go test ./table -run ^ -bench ^BenchmarkPlanFilesWithRowLimit -benchmem -benchtime=100ms -count=1` `no-limit` is the full-planning control. `limit-1` means `UseRowLimit(1)`. | Manifests | No limit | `UseRowLimit(1)` | | ---: | ---: | ---: | | 100 | 23.5 ms/op, 27.5 MB/op | 403 µs/op, 514 KB/op | | 1,000 | 188.5 ms/op, 273.2 MB/op | 785 µs/op, 910 KB/op | | 10,000 | 1.73 s/op, 2.73 GB/op | 4.71 ms/op, 5.19 MB/op | The 10,000-manifest case is about **368x faster** with about **525x fewer allocated bytes**. 📈 ## Tests - `go test ./table/... -count=1` - `go test -race ./table -run 'Test(LimitManifestListByRows|CanLimitLocalPlanning|PlanFilesLocalUsesRowLimitToStopOpeningManifests)$' -count=1` - `go test ./... -run '^$' -count=1` - `go vet ./table` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
