JingsongLi commented on code in PR #1145:
URL: https://github.com/apache/paimon-rust/pull/1145#discussion_r4435278357
##########
crates/paimon/src/table/de_vector_read.rs:
##########
@@ -639,11 +639,14 @@ async fn scalar_matched_row_ids(
let Some(scanner) = scanner else {
return Ok(None);
};
- let Some((ranges, exact)) =
scanner.matching_ranges_with_exactness(filter).await? else {
+ let Some((ranges, exact, candidate_ranges)) =
+ scanner.matching_ranges_with_exactness(filter).await?
+ else {
return Ok(None);
};
let ranges = intersect_sorted_ranges(&ranges, vector_ranges);
- if exact {
+ // Candidates outside the rows this search ranks do not make it inexact.
+ if exact && intersect_sorted_ranges(&candidate_ranges, &ranges).is_empty()
{
Review Comment:
[P2] Preserve exact BTree hits alongside Bitmap candidates
The new condition downgrades the entire scalar answer when any
candidate-only range remains. With
`global-index.filter.refine-from-data=false`, the following branch then returns
an empty set, including disjoint rows already proven by an exact BTree index.
The new scanner contract distinguishes candidate-only rows; disabling
refinement should exclude those rows, rather than the exact remainder.
An independent public write/build-index/search/read probe writes rows 0/1
with f=0/1, builds Bitmap(f), appends rows 2/3 with f=2/3 and builds BTree(f),
then builds IVF-flat over all embeddings. For `f > 1.5`, query [1,0], top-1 is
exact BTree row 2; Bitmap rows 0/1 are only candidates and are farther away.
This head returns [] in FAST, FULL and DETAIL with refinement disabled, while
all enabled-refinement controls return [2]. The identical probe passes all six
combinations on exact merge-base fb5e8f7e.
When `exact` is true, retain `ranges - candidate_ranges` if refinement is
disabled; otherwise refine candidate rows without losing exact ones. The
full-text consumer at filter.rs:197 has the same all-or-nothing classification
and should be checked as part of the fix. Please cover mixed candidate/exact
shards within the vector-covered domain.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]