JingsongLi commented on code in PR #1475:
URL: https://github.com/apache/paimon-rust/pull/1475#discussion_r4643414029


##########
crates/paimon/src/table/read_execution.rs:
##########
@@ -113,7 +113,15 @@ pub(super) fn execute(
     limit: Option<usize>,
     read: SplitReader,
 ) -> ArrowRecordBatchStream {
-    let streams = split_groups(splits, parallelism)
+    // Ordered output needs a sliding window of individual splits. Grouping
+    // several splits into one stream leaves the later splits in each group
+    // unopened until every earlier split in that group has been emitted.
+    let groups = if ordered && parallelism > 1 {

Review Comment:
   **[P2] Defer speculative split errors until an ordered LIMIT needs that 
split**
   
   With four equal-size splits, parallelism 2 and LIMIT 1 over a filtered read, 
the old grouping starts splits 0 and 2. The new window starts 0 and 1. When 
split 0 can return the only row requested but is delayed and split 1 errors on 
open, the merge surfaces split 1's error before returning any rows. I 
reproduced this through `execute` with a gated first split and a failing second 
split. The identical test passes on `ab2eaef4` and fails on `b0b1eaad` in 
repeated runs.
   
   For a finite ordered limit, buffer failures from speculative splits in 
output order and surface them when the scan reaches that split. Once the limit 
is satisfied, discard unused split failures with the remaining work. Unlimited 
ordered scans can retain their existing fail-fast behavior.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to