XiaoHongbo-Hope opened a new pull request, #10868: URL: https://github.com/apache/paimon/pull/10868
## Purpose Temporal alignment plans each input and reads each source payload serially, adding their remote I/O latency. Add opt-in `to_arrow_batch_reader(parallelism=4)` to overlap independent source planning and per-batch payload reads while preserving input/output order. The default remains serial. Concurrency is bounded by the requested parallelism and source count. Each batch finishes before the next is consumed. Failure cancels queued work and joins running tasks before propagating the error; existing snapshot, authorization and projection paths remain in use. Per-table reader concurrency still applies, and custom catalog/auth callbacks must support concurrent calls when enabled. Depends on #10846. The additional change is the commit titled `[python] Overlap independent temporal source planning and reads`; inherited changes will disappear from the diff once #10846 is merged and this branch is rebased. ## Validation - Real native extension: 90 temporal tests and 160 subtests passed. - Compatibility reader path: 89 passed, 1 native-only skip, 160 subtests passed during implementation. - Added real-table checks for serial/parallel output equality, bounded concurrent reads, independent planning, worker cleanup on read/planning failures, and argument validation. - Flake8 with repository configuration and `git diff --check` passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
