jianguotian opened a new pull request, #851: URL: https://github.com/apache/paimon-rust/pull/851
## Purpose The Rust Mosaic reader currently opens row groups serially. Opening a group performs dependent range reads and decompression, so object-store latency remains on the critical path even after manifest and row-selection pruning. ## Changes - Plan row groups after row-selection and statistics pruning, then prefetch only the survivors. - Open up to 8 row groups concurrently while preserving file order in emitted batches. - Bound decoded data held ahead with a 64 MiB estimate based on projected field types and row counts. - Always admit the head group so an undersized byte budget cannot stall progress. - Keep residual predicate evaluation and batch emission ordered on the consumer thread. - Join all scheduled work on cancellation, errors, and panics so no storage reads outlive the Mosaic reader. The defaults match Java's `mosaic.read.prefetch-row-groups=8` and `mosaic.read.prefetch-max-bytes=64mb`. Rust's format-reader factory does not yet expose format-scoped table options, so this PR establishes the bounded behavior with those defaults; option plumbing can remain an independent API change. ## Verification - `cargo test -p paimon --lib`: 2,818 passed, 2 ignored - `cargo clippy -p paimon --lib --tests -- -D warnings` - A delayed storage fixture verifies overlapping row-group I/O and stable output order. - A one-byte budget regression verifies fallback to one mandatory in-flight group. ## Related Java work - apache/paimon#9740 - apache/paimon#9742 - apache/paimon#9744 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
