fallintoplace opened a new pull request, #1939: URL: https://github.com/apache/iceberg-go/pull/1939
## What changed - **Split large local Parquet files** at their recorded row-group offsets. - Added `read.split.target-size` with a **128 MB default**. - Parquet reads now keep only row groups inside each task range. - Keep original file row positions for `_row_id`, positional deletes, and deletion vectors. - Keep scan report file and delete metrics at the original data-file level. ## Why A table with a few large Parquet files currently gives the scan one task per file. This lets the existing workers read independent row groups in parallel. Files without safe split offsets stay as one task. ## Benchmark 📈 Apple M1 Pro, 8 row groups, 32768 rows per group, in-memory Parquet, 8 workers. Three runs with `-benchtime=2s -count=3`. - One task: **6.21 to 6.80 ms/op**, 29.7 MB/op, about 2,759 allocs/op - Row-group tasks: **4.44 to 5.50 ms/op**, 69.1 MB/op, about 7,125 allocs/op - About **30% faster** in this workload. Splitting uses more allocations because each task opens its own reader. ## Tests - `go test ./table/...` - `go test ./catalog/rest/...` - `go vet ./table/...` - `go test -race ./table -run TestPlanFilesSplitsLargeParquetFileAndReadsEachRowOnce -count=1` - `go test ./table -run ^$ -bench BenchmarkArrowScanLargeParquetFileSplitTasks -benchmem -benchtime=2s -count=3` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
