JingsongLi commented on code in PR #757:
URL: https://github.com/apache/paimon-rust/pull/757#discussion_r3900604884
##########
crates/paimon/src/table/vector_search_builder.rs:
##########
@@ -1061,6 +1175,15 @@ async fn plan_and_search_pk_candidates_batch(
}
None => None,
};
+ // Fold the plan's own positional restriction into the same allow-list. A
plan
+ // built from engine-supplied bucket splits carries the physical positions
each
+ // file is limited to; a plan read from the index manifest carries none.
Both
+ // sides list what is permitted, so combining them is an intersection.
+ let residual_by_split = intersect_row_allow_lists(
Review Comment:
[P1] Apply the supplied physical ranges before scanning residuals
For an engine-supplied bucket split, `plan.physical_row_ranges_by_split` can
restrict a large data file to a very small range. When a data predicate is also
present, the code calls `residual_positions_by_file` first; that function
invokes `read_single_file_stream(..., None, None)` and scans every physical row
in each active file. Only here, afterward, are positions outside the supplied
range discarded. Thus a 10-row split over a billion-row file turns into a
full-file read for every residual-filtered vector query, defeating the
distributed split and creating an extreme I/O/latency regression. Java
evaluates the residual through an `IndexedSplit` built from the candidate
ranges. Please pass the physical allow-list into residual evaluation, skip
empty files, and read/evaluate only those ranges; a counting-reader test should
assert that rows outside the engine range are never read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
