JingsongLi commented on code in PR #1145:
URL: https://github.com/apache/paimon-rust/pull/1145#discussion_r4492060335


##########
crates/paimon/src/table/global_index_scanner/evaluator.rs:
##########
@@ -749,6 +828,11 @@ impl GlobalIndexScanner {
             crate::table::merge_row_ranges(all_ranges),
             coverage(),
             fully_evaluated,
+            if bitmap_superset_ops && !bitmap_ranges.is_empty() {
+                crate::table::merge_row_ranges(bitmap_ranges)

Review Comment:
   [P2] Preserve exact BTree hits when Bitmap coverage overlaps
   
   `bitmap_ranges` includes rows that the same leaf's BTree result has already 
proven to match. Different index types are allowed to cover the same field and 
row range, so these are not necessarily candidate-only rows. With 
`global-index.filter.refine-from-data=false`, both search consumers subtract 
these ranges and discard the proven matches as well. The existing mixed-shard 
tests cover disjoint Bitmap/BTree rows; this case has overlapping coverage.
   
   I reproduced this through public write, commit, index-build, search and 
result-read APIs:
   
   1. Create an append table with `id INT`, `f DOUBLE`, `text STRING` and 
`embedding ARRAY<FLOAT>`. Set `bucket=-1`, `row-tracking.enabled=true`, 
`data-evolution.enabled=true` and `global-index.enabled=true`.
   2. Write and commit these two rows:
   
      | id | f | text | embedding |
      | --- | --- | --- | --- |
      | 0 | 1.0 | other | [0.0, 1.0] |
      | 1 | 2.0 | paimon | [1.0, 0.0] |
   
   3. Without appending any more data, build Bitmap(f), then BTree(f), then 
IVF-flat(embedding) with `dimension=2`, `nlist=1`, `distance.metric=l2`, and a 
full-text index on `text`.
   4. Set `global-index.filter.refine-from-data=false`. Use 
`pb.greater_than("f", Datum::Double(1.5))` as the filter. Run vector Top-1 for 
query `[1.0, 0.0]` and full-text Top-1 for query `"paimon"`, then read each 
result and collect `id`.
   5. Repeat with `vector-index.search-mode`, `scalar-index.search-mode` and 
`full-text-index.search-mode` set to `fast`, `full` and `detail`.
   
   Both searches should return `[1]`. On head 
`8d30fada4e695ce253b8c93f611c99d72af17f73`, both return `[]` in all three 
modes. BTree proves row 1 matches, but Bitmap returns both non-null rows as 
candidates, leaving `candidate_ranges=[0,1]`; subtracting those candidates 
removes row 1 too.
   
   The identical probe passes on exact base/merge-base 
`7a4f3bc89ad0d1ef814db82af1f5d2fba17fb12c`:
   
   | Scalar indexes | Refine from data | Base: vector / full-text | Head: 
vector / full-text |
   | --- | --- | --- | --- |
   | BTree only | false or true | [1] / [1] | [1] / [1] |
   | Bitmap + BTree, same rows | true | [1] / [1] | [1] / [1] |
   | Bitmap + BTree, same rows | false | [1] / [1] | [] / [] |
   
   Each result above holds in FAST, FULL and DETAIL: all 24 combinations pass 
on base; 6 fail on head. This probe contains no NULLs, NaNs or predicate 
negation.
   
   Please preserve fully evaluated exact hits when combining readers for a 
leaf, and remove those proven rows from the Bitmap candidate ranges, as the OR 
path now does. Keep partial-evaluation/fallback status in that decision, and 
add an overlapping-coverage regression for both vector and full-text consumers.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to