JunRuiLee commented on code in PR #757:
URL: https://github.com/apache/paimon-rust/pull/757#discussion_r3901509310
##########
crates/paimon/src/table/vector_search_builder.rs:
##########
@@ -1061,6 +1175,15 @@ async fn plan_and_search_pk_candidates_batch(
}
None => None,
};
+ // Fold the plan's own positional restriction into the same allow-list. A
plan
+ // built from engine-supplied bucket splits carries the physical positions
each
+ // file is limited to; a plan read from the index manifest carries none.
Both
+ // sides list what is permitted, so combining them is an intersection.
+ let residual_by_split = intersect_row_allow_lists(
Review Comment:
Fixed in 3b9ba6f. The plan now carries the selection as normalized
`Vec<RowRange>` rather than materialized positions, and both paths that touch a
data file read through those ranges — the residual, and the exact fallback,
which you did not mention but has the same defect and does not even need a
predicate to hit it. Positions are recovered by walking the selection in step
with the emitted rows and the two are checked against each other, so a read
that overshoots fails loudly instead of quietly returning a filtered answer. A
file the plan lists no rows for is registered empty without being opened.
`DataFileReader` already had a file-local ranges path — it coalesced
positions into ranges internally — so it grew a ranges entry point rather than
a new one. I did not use the generic `row_ranges` argument: that one translates
through `first_row_id`, which is not the file-local space these ranges are in.
On the counting test: it is at the format layer, where the reading happens.
A selection inside one row group requests strictly fewer byte ranges and fewer
bytes than a full read. One caveat I should state rather than let you find —
what the selection saves depends on the format. Mosaic skips a row group before
touching its column data, parquet skips pages through the offset index, `.row`
prunes blocks, but Avro loads the whole file and deserializes every record
before applying the selection, so for an Avro data file the full read remains.
Java allows `file.format=avro` for a vector column, so that is reachable.
Making Avro prune physically needs a block-aware reader; I left it stated
rather than half-done.
The intersection you pointed at stays. It cannot remove anything once the
residual was evaluated over the same ranges, which is the invariant it now
records.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
