JingsongLi commented on code in PR #1145:
URL: https://github.com/apache/paimon-rust/pull/1145#discussion_r4492060335
##########
crates/paimon/src/table/global_index_scanner/evaluator.rs:
##########
@@ -749,6 +828,11 @@ impl GlobalIndexScanner {
crate::table::merge_row_ranges(all_ranges),
coverage(),
fully_evaluated,
+ if bitmap_superset_ops && !bitmap_ranges.is_empty() {
+ crate::table::merge_row_ranges(bitmap_ranges)
Review Comment:
[P2] Preserve exact BTree hits when Bitmap coverage overlaps
`bitmap_ranges` includes rows that the same leaf's BTree result has already
proven to match. Different index types are allowed to cover the same field and
row range, so these are not necessarily candidate-only rows. With
`global-index.filter.refine-from-data=false`, both search consumers subtract
these ranges and discard the proven matches as well. The existing mixed-shard
tests cover disjoint Bitmap/BTree rows; this case has overlapping coverage.
I reproduced this through public write, commit, index-build, search and
result-read APIs:
1. Create an append table with `id INT`, `f DOUBLE`, `text STRING` and
`embedding ARRAY<FLOAT>`. Set `bucket=-1`, `row-tracking.enabled=true`,
`data-evolution.enabled=true` and `global-index.enabled=true`.
2. Write and commit these two rows:
| id | f | text | embedding |
| --- | --- | --- | --- |
| 0 | 1.0 | other | [0.0, 1.0] |
| 1 | 2.0 | paimon | [1.0, 0.0] |
3. Without appending any more data, build Bitmap(f), then BTree(f), then
IVF-flat(embedding) with `dimension=2`, `nlist=1`, `distance.metric=l2`, and a
full-text index on `text`.
4. Set `global-index.filter.refine-from-data=false`. Use
`pb.greater_than("f", Datum::Double(1.5))` as the filter. Run vector Top-1 for
query `[1.0, 0.0]` and full-text Top-1 for query `"paimon"`, then read each
result and collect `id`.
5. Repeat with `vector-index.search-mode`, `scalar-index.search-mode` and
`full-text-index.search-mode` set to `fast`, `full` and `detail`.
Both searches should return `[1]`. On head
`8d30fada4e695ce253b8c93f611c99d72af17f73`, both return `[]` in all three
modes. BTree proves row 1 matches, but Bitmap returns both non-null rows as
candidates, leaving `candidate_ranges=[0,1]`; subtracting those candidates
removes row 1 too.
The identical probe passes on exact base/merge-base
`7a4f3bc89ad0d1ef814db82af1f5d2fba17fb12c`:
| Scalar indexes | Refine from data | Base: vector / full-text | Head:
vector / full-text |
| --- | --- | --- | --- |
| BTree only | false or true | [1] / [1] | [1] / [1] |
| Bitmap + BTree, same rows | true | [1] / [1] | [1] / [1] |
| Bitmap + BTree, same rows | false | [1] / [1] | [] / [] |
Each result above holds in FAST, FULL and DETAIL: all 24 combinations pass
on base; 6 fail on head. This probe contains no NULLs, NaNs or predicate
negation.
Please preserve fully evaluated exact hits when combining readers for a
leaf, and remove those proven rows from the Bitmap candidate ranges, as the OR
path now does. Keep partial-evaluation/fallback status in that decision, and
add an overlapping-coverage regression for both vector and full-text consumers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]