guptashivam04 opened a new pull request, #70678:
URL: https://github.com/apache/doris/pull/70678

   ### What problem does this PR solve?
   
   Issue Number: None
   Related PR: #70668 (split from it)
   PR Type: Improvement
   
   Problem Summary:
   
   A SEARCH next to a selective filter outside it, as candidate search writes 
it:
   
   ```sql
   SELECT candidate_id FROM candidates
   WHERE account_id = 12345
     AND search('(title:"software engineer" OR title:"cloud engineer") AND 
skill:ANY(python sql)')
   ```
   
   evaluated the SEARCH over the whole segment, every account included, and 
only then intersected the account's rows. `SegmentIterator` hides its candidate 
rows from roots that combine three-valued bitmaps, and SEARCH is one of them, 
so the rows `account_id = 12345` had already narrowed the scan to were never 
seen by SEARCH.
   
   ### What is changed?
   
   - A SEARCH that is a whole filtering conjunct gets the scan's candidate rows 
at its root, through a new `IndexQueryContext::search_root_candidates`. Its 
leaves still see none, so how they combine three-valued results does not 
change. A SEARCH used as a virtual column, or inside a compound predicate, is 
not affected.
   - An unscored SEARCH collects its tree within those rows: a root that lists 
its rows lists them within the candidates, any other root is read through its 
scorer within them (`collect_truth_set` already handles both sparse and dense 
candidates).
   - The conjunct's result is only read within the candidates, so a two-valued 
answer outside them is never used. A result computed this way is partial, so it 
is never put into the DSL query cache.
   - Scored SEARCH is unchanged: it keeps every match, and scores do not depend 
on the candidates.
   
   The candidate rows come from whatever narrowed the scan before SEARCH runs: 
an inverted index on `account_id`, zone maps and the short key, the delete 
bitmap.
   
   ### Numbers
   
   2M rows, 3 BEs, inverted index on `account_id`, caches off, median of 3 
runs. Rows and scores are byte-identical on 19 SEARCH queries (unscored and 
scored, lucene and standard mode, top-k), with the inverted index query cache 
off and on.
   
   | query | before | after |
   |---|---|---|
   | ranking query of 16 SEARCH scans, account with 648 rows (0.03%) | 1648 ms, 
7.9 s CPU | 1008 ms, 5.1 s CPU |
   | same, account with 50K rows (2.5%) | 2002 ms, 8.8 s CPU | 1763 ms, 8.5 s 
CPU |
   | `score()` queries (6 shapes, incl. top-k) | | unchanged |
   
   A SEARCH whose root is an AND of OR groups gains little here, because those 
groups still evaluate their children over the segment; a follow-up lets nested 
groups read only the rows they are handed.
   
   ### Release note
   
   None
   
   ### Check List (For Author)
   
   - Test
       - [x] Regression test
           - `search/test_search_outer_filter_candidates`: on a table with an 
inverted index on the account and on one with the account leading the sort key, 
the rows of `account = 7 AND search(...)` against the full SEARCH filtered 
afterwards, in lucene and standard mode, with AND, OR and NOT; scores against 
the full SEARCH; with the query cache on, the filtered query first, so a 
partial result would reach the unfiltered one if it were cached.
           - All `search/` and `inverted_index_p0` suites pass (298), except 3 
that fail the same way without this change on the same machine (custom analyzer 
dictionaries and replica setup missing locally).
       - [x] Unit Test
           - 
`BooleanTruthContractTest.CollectionWithinCandidatesKeepsTheirMatches`: an 
unscored collection given candidates keeps exactly the matches among them, for 
a weight that lists its rows and for one read through its scorer.
           - Existing suites pass (607 tests): SEARCH function and NULL 
handling, BM25, compound predicate scores, segment iterator index application 
and candidate pushdown, SNII readers, collectors.
   - Behavior changed: No. Rows and scores are the same.
   - Does this need documentation: No
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to