This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
     new 9f7ce885ec5 [opt](search) Start a SEARCH conjunct from the rows the 
scan's other filters kept (#70678)
9f7ce885ec5 is described below

commit 9f7ce885ec5db6431bd418de69a48c18e3f1806b
Author: guptashivam04 <[email protected]>
AuthorDate: Thu Nov 19 22:55:27 2026 +0530

    [opt](search) Start a SEARCH conjunct from the rows the scan's other 
filters kept (#70678)
    
    ### What problem does this PR solve?
    
    Issue Number: None
    Related PR: #70668 (split from it)
    PR Type: Improvement
    
    Problem Summary:
    
    A SEARCH next to a selective filter outside it, as candidate search
    writes it:
    
    ```sql
    SELECT candidate_id FROM candidates
    WHERE account_id = 12345
      AND search('(title:"software engineer" OR title:"cloud engineer") AND 
skill:ANY(python sql)')
    ```
    
    evaluated the SEARCH over the whole segment, every account included, and
    only then intersected the account's rows. `SegmentIterator` hides its
    candidate rows from roots that combine three-valued bitmaps, and SEARCH
    is one of them, so the rows `account_id = 12345` had already narrowed
    the scan to were never seen by SEARCH.
    
    ### What is changed?
    
    - A SEARCH that is a whole filtering conjunct gets the scan's candidate
    rows at its root, through a new
    `IndexQueryContext::search_root_candidates`. Its leaves still see none,
    so how they combine three-valued results does not change. A SEARCH used
    as a virtual column, or inside a compound predicate, is not affected.
    - An unscored SEARCH collects its tree within those rows: a root that
    lists its rows lists them within the candidates, any other root is read
    through its scorer within them (`collect_truth_set` already handles both
    sparse and dense candidates).
    - The conjunct's result is only read within the candidates, so a
    two-valued answer outside them is never used. A result computed this way
    is partial, so it is never put into the DSL query cache.
    - Scored SEARCH is unchanged: it keeps every match, and scores do not
    depend on the candidates.
    
    The candidate rows come from whatever narrowed the scan before SEARCH
    runs: an inverted index on `account_id`, zone maps and the short key,
    the delete bitmap.
    
    ### Numbers
    
    2M rows, 3 BEs, inverted index on `account_id`, caches off, median of 3
    runs. Rows and scores are byte-identical on 19 SEARCH queries (unscored
    and scored, lucene and standard mode, top-k), with the inverted index
    query cache off and on.
    
    | query | before | after |
    |---|---|---|
    | ranking query of 16 SEARCH scans, account with 648 rows (0.03%) | 1648
    ms, 7.9 s CPU | 1008 ms, 5.1 s CPU |
    | same, account with 50K rows (2.5%) | 2002 ms, 8.8 s CPU | 1763 ms, 8.5
    s CPU |
    | `score()` queries (6 shapes, incl. top-k) | | unchanged |
    
    A SEARCH whose root is an AND of OR groups gains little here, because
    those groups still evaluate their children over the segment; a follow-up
    lets nested groups read only the rows they are handed.
    
    ### Release note
    
    None
    
    ### Check List (For Author)
    
    - Test
        - [x] Regression test
    - `search/test_search_outer_filter_candidates`: on a table with an
    inverted index on the account and on one with the account leading the
    sort key, the rows of `account = 7 AND search(...)` against the full
    SEARCH filtered afterwards, in lucene and standard mode, with AND, OR
    and NOT; scores against the full SEARCH; with the query cache on, the
    filtered query first, so a partial result would reach the unfiltered one
    if it were cached.
    - All `search/` and `inverted_index_p0` suites pass (298), except 3 that
    fail the same way without this change on the same machine (custom
    analyzer dictionaries and replica setup missing locally).
        - [x] Unit Test
    -
    `BooleanTruthContractTest.CollectionWithinCandidatesKeepsTheirMatches`:
    an unscored collection given candidates keeps exactly the matches among
    them, for a weight that lists its rows and for one read through its
    scorer.
    - Existing suites pass (607 tests): SEARCH function and NULL handling,
    BM25, compound predicate scores, segment iterator index application and
    candidate pushdown, SNII readers, collectors.
    - Behavior changed: No. Rows and scores are the same.
    - Does this need documentation: No
    
    ---------
    
    Co-authored-by: Shivam Gupta <[email protected]>
---
 be/src/exprs/function/search/function_search.cpp   |  11 +-
 be/src/storage/index/index_query_context.h         |   7 +
 .../query/engine/collect/doc_set_collector.cpp     |  10 +-
 .../query/engine/collect/doc_set_collector.h       |   3 +-
 be/src/storage/segment/segment_iterator.cpp        |  21 +-
 .../query/engine/boolean_truth_contract_test.cpp   |  94 +++
 .../search/test_search_outer_filter_candidates.out | 741 +++++++++++++++++++++
 .../test_search_outer_filter_candidates.groovy     | 107 +++
 8 files changed, 986 insertions(+), 8 deletions(-)

diff --git a/be/src/exprs/function/search/function_search.cpp 
b/be/src/exprs/function/search/function_search.cpp
index 6d47098e5f6..e4e953c70dd 100644
--- a/be/src/exprs/function/search/function_search.cpp
+++ b/be/src/exprs/function/search/function_search.cpp
@@ -553,6 +553,12 @@ Status FunctionSearch::_evaluate_search_param(
         top_k = index_query_context->query_limit;
     }
 
+    // A SEARCH that is a whole conjunct reads only the rows the scan still 
holds. Scored, it
+    // keeps every match, as the scores of the rows it returns do not depend 
on them.
+    const roaring::Roaring* root_candidates = !enable_scoring && 
index_query_context != nullptr
+                                                      ? 
index_query_context->search_root_candidates
+                                                      : nullptr;
+
     auto weight = root_query->weight(enable_scoring);
     if (!weight) {
         LOG(WARNING) << "search: Failed to build query weight";
@@ -589,7 +595,7 @@ Status FunctionSearch::_evaluate_search_param(
             query_v2::collect_multi_segment_doc_set(
                     weight, exec_ctx, root_binding_key, roaring,
                     index_query_context ? 
index_query_context->collection_similarity : nullptr,
-                    enable_scoring);
+                    enable_scoring, nullptr, root_candidates);
         }
     }
 
@@ -599,7 +605,8 @@ Status FunctionSearch::_evaluate_search_param(
     bitmap_result =
             InvertedIndexResultBitmap(std::move(roaring), 
std::make_shared<roaring::Roaring>());
 
-    if (dsl_cache && cache_usable) {
+    // A result read within candidates is partial and stays out of the cache.
+    if (dsl_cache && cache_usable && root_candidates == nullptr) {
         insert_dsl_result(dsl_cache, seg_prefix, dsl_sig, bitmap_result);
     }
 
diff --git a/be/src/storage/index/index_query_context.h 
b/be/src/storage/index/index_query_context.h
index 32409627a76..52a360b8a3a 100644
--- a/be/src/storage/index/index_query_context.h
+++ b/be/src/storage/index/index_query_context.h
@@ -73,6 +73,13 @@ struct IndexQueryContext {
     // evaluation.
     const roaring::Roaring* candidate_rows = nullptr;
 
+    // The same candidate rows, for a SEARCH that is a whole conjunct on its 
own. Its leaves
+    // combine three-valued bitmaps, so SegmentIterator hides candidate_rows 
from them and hands
+    // the rows here instead: SEARCH starts its root from them, and its 
two-valued result is
+    // read only within them. Set and reset like candidate_rows, and a result 
computed from
+    // them is partial in the same way: it never goes into the query cache.
+    const roaring::Roaring* search_root_candidates = nullptr;
+
     // ---- Reply direction: fields a READER writes and the CALLER reads back 
----
     //
     // A reader writes them on the context it was given, never on a copy: a 
copy's reply would
diff --git 
a/be/src/storage/index/inverted/query/engine/collect/doc_set_collector.cpp 
b/be/src/storage/index/inverted/query/engine/collect/doc_set_collector.cpp
index 3c9521081ea..51a4d5554f4 100644
--- a/be/src/storage/index/inverted/query/engine/collect/doc_set_collector.cpp
+++ b/be/src/storage/index/inverted/query/engine/collect/doc_set_collector.cpp
@@ -17,6 +17,7 @@
 
 #include "storage/index/inverted/query/engine/collect/doc_set_collector.h"
 
+#include "common/check.h"
 #include "storage/index/inverted/query/engine/complete_null_bitmap.h"
 
 namespace doris::segment_v2::inverted_index::query_v2 {
@@ -25,12 +26,15 @@ void collect_multi_segment_doc_set(const WeightPtr& weight, 
const QueryExecution
                                    const std::string& binding_key,
                                    const std::shared_ptr<roaring::Roaring>& 
roaring,
                                    const CollectionSimilarityPtr& similarity, 
bool enable_scoring,
-                                   roaring::Roaring* null_rows) {
+                                   roaring::Roaring* null_rows,
+                                   const roaring::Roaring* candidates) {
     if (context.segment_num_rows == 0) {
         return;
     }
+    // Candidates restrict an unscored collection only; the caller reads its 
rows within them.
+    DORIS_CHECK(candidates == nullptr || !enable_scoring);
     if (!enable_scoring && weight->lists_rows(context, binding_key)) {
-        auto rows = weight->listed_rows(context, binding_key, nullptr);
+        auto rows = weight->listed_rows(context, binding_key, candidates);
         *roaring |= rows.true_rows;
         if (null_rows != nullptr) {
             *null_rows |= rows.null_rows;
@@ -49,6 +53,8 @@ void collect_multi_segment_doc_set(const WeightPtr& weight, 
const QueryExecution
     }
     if (enable_scoring && similarity != nullptr) {
         *roaring |= collect_scored_rows(scorer, 0, *similarity);
+    } else if (candidates != nullptr) {
+        *roaring |= collect_truth_set(scorer, context.null_resolver, 
candidates).true_rows;
     } else {
         collect_true_rows(scorer, roaring.get());
     }
diff --git 
a/be/src/storage/index/inverted/query/engine/collect/doc_set_collector.h 
b/be/src/storage/index/inverted/query/engine/collect/doc_set_collector.h
index ae26cba6429..dd36c92a5cb 100644
--- a/be/src/storage/index/inverted/query/engine/collect/doc_set_collector.h
+++ b/be/src/storage/index/inverted/query/engine/collect/doc_set_collector.h
@@ -32,6 +32,7 @@ void collect_multi_segment_doc_set(const WeightPtr& weight, 
const QueryExecution
                                    const std::string& binding_key,
                                    const std::shared_ptr<roaring::Roaring>& 
roaring,
                                    const CollectionSimilarityPtr& similarity, 
bool enable_scoring,
-                                   roaring::Roaring* null_rows = nullptr);
+                                   roaring::Roaring* null_rows = nullptr,
+                                   const roaring::Roaring* candidates = 
nullptr);
 
 } // namespace doris::segment_v2::inverted_index::query_v2
diff --git a/be/src/storage/segment/segment_iterator.cpp 
b/be/src/storage/segment/segment_iterator.cpp
index 6edfbb7b18d..89f5777b067 100644
--- a/be/src/storage/segment/segment_iterator.cpp
+++ b/be/src/storage/segment/segment_iterator.cpp
@@ -909,6 +909,10 @@ static bool can_consume_candidate_rows(const 
VExprContextSPtr& expr_ctx) {
         DORIS_CHECK(virtual_expr != nullptr);
         effective_root = virtual_expr.get();
     }
+    // A filtering SEARCH starts from the candidate rows at its root.
+    if (!root->is_virtual_slot_ref() && root->node_type() == 
TExprNodeType::SEARCH_EXPR) {
+        return true;
+    }
     if (effective_root->node_type() != TExprNodeType::MATCH_PRED) {
         return false;
     }
@@ -1505,7 +1509,10 @@ Status SegmentIterator::_apply_index_expr() {
     // a compound predicate or a SEARCH, which combines its own leaf bitmaps 
the
     // same way, are therefore evaluated without the candidate; the top-level
     // single-predicate consumption stays exact within the candidate.
-    auto evaluate_with_candidate_policy = [&](const VExprContextSPtr& 
expr_ctx) {
+    // A filtering conjunct that is a SEARCH by itself still gets the 
candidate, at its root
+    // only: SEARCH answers two-valued, and the conjunct reads its result only 
within the
+    // candidate.
+    auto evaluate_with_candidate_policy = [&](const VExprContextSPtr& 
expr_ctx, bool filters) {
         // Earlier expression conjuncts may have crossed the engage threshold.
         // Refresh for both subsequent conjuncts and virtual-column 
projections.
         _refresh_candidate_pushdown();
@@ -1522,13 +1529,21 @@ Status SegmentIterator::_apply_index_expr() {
                               _index_query_context->candidate_rows != nullptr 
&&
                               combines_index_results(*effective_root);
         const roaring::Roaring* saved = suppress ? 
_index_query_context->candidate_rows : nullptr;
+        const bool search_root = suppress && filters && 
!root->is_virtual_slot_ref() &&
+                                 root->node_type() == 
TExprNodeType::SEARCH_EXPR;
         if (suppress) {
             _index_query_context->candidate_rows = nullptr;
         }
+        if (search_root) {
+            _index_query_context->search_root_candidates = saved;
+        }
         Status st = expr_ctx->evaluate_inverted_index(num_rows());
         if (suppress) {
             _index_query_context->candidate_rows = saved;
         }
+        if (search_root) {
+            _index_query_context->search_root_candidates = nullptr;
+        }
         return st;
     };
 
@@ -1555,7 +1570,7 @@ Status SegmentIterator::_apply_index_expr() {
         // answer approximately then knows its work will be used. The virtual 
column loop
         // further down consumes none, and deliberately names none.
         expr_ctx->set_approx_index_result_consumer(expr_ctx->root().get());
-        if (Status st = evaluate_with_candidate_policy(expr_ctx); !st.ok()) {
+        if (Status st = evaluate_with_candidate_policy(expr_ctx, 
/*filters=*/true); !st.ok()) {
             if (_downgrade_without_index(st) || st.code() == 
ErrorCode::NOT_IMPLEMENTED_ERROR) {
                 continue;
             } else {
@@ -1605,7 +1620,7 @@ Status SegmentIterator::_apply_index_expr() {
             if (expr_ctx->get_index_context() == nullptr) {
                 continue;
             }
-            if (Status st = evaluate_with_candidate_policy(expr_ctx); 
!st.ok()) {
+            if (Status st = evaluate_with_candidate_policy(expr_ctx, 
/*filters=*/false); !st.ok()) {
                 if (_downgrade_without_index(st) || st.code() == 
ErrorCode::NOT_IMPLEMENTED_ERROR) {
                     continue;
                 } else {
diff --git 
a/be/test/storage/index/inverted/query/engine/boolean_truth_contract_test.cpp 
b/be/test/storage/index/inverted/query/engine/boolean_truth_contract_test.cpp
index 41503b945bd..acff6829085 100644
--- 
a/be/test/storage/index/inverted/query/engine/boolean_truth_contract_test.cpp
+++ 
b/be/test/storage/index/inverted/query/engine/boolean_truth_contract_test.cpp
@@ -30,6 +30,7 @@
 #include "storage/index/inverted/query/engine/bit_set_query/bit_set_scorer.h"
 #include 
"storage/index/inverted/query/engine/boolean_query/boolean_query_builder.h"
 #include "storage/index/inverted/query/engine/boolean_query/operator.h"
+#include "storage/index/inverted/query/engine/collect/doc_set_collector.h"
 #include "storage/index/inverted/query/engine/null_bitmap_test_util.h"
 
 namespace doris::segment_v2::inverted_index::query_v2 {
@@ -559,5 +560,98 @@ TEST(BooleanTruthContractTest, 
OptionalShouldBesideARequiredClauseIsTwoValued) {
     }
 }
 
+// Lists its rows itself, like a phrase root, and records the candidates each 
listing gets.
+class ListingWeight final : public Weight {
+public:
+    ListingWeight(TruthRows rows, 
std::shared_ptr<std::vector<roaring::Roaring>> listed)
+            : _rows(std::move(rows)), _listed(std::move(listed)) {}
+    ScorerPtr scorer(const QueryExecutionContext& /*context*/) override {
+        ADD_FAILURE() << "a weight that lists its rows is not read through a 
scorer";
+        return nullptr;
+    }
+    bool lists_rows(const QueryExecutionContext& /*context*/,
+                    const std::string& /*binding_key*/) const override {
+        return true;
+    }
+    index_query::TruthSet listed_rows(const QueryExecutionContext& /*context*/,
+                                      const std::string& /*binding_key*/,
+                                      const roaring::Roaring* candidates) 
override {
+        EXPECT_NE(candidates, nullptr);
+        index_query::TruthSet result {*_rows.true_rows, *_rows.null_rows};
+        if (candidates != nullptr) {
+            _listed->push_back(*candidates);
+            result.true_rows &= *candidates;
+            result.null_rows &= *candidates;
+        }
+        return result;
+    }
+
+private:
+    TruthRows _rows;
+    std::shared_ptr<std::vector<roaring::Roaring>> _listed;
+};
+
+class ListingQuery final : public Query {
+public:
+    ListingQuery(TruthRows rows, 
std::shared_ptr<std::vector<roaring::Roaring>> listed)
+            : _rows(std::move(rows)), _listed(std::move(listed)) {}
+    WeightPtr weight(bool /*enable_scoring*/) override {
+        return std::make_shared<ListingWeight>(_rows, _listed);
+    }
+
+private:
+    TruthRows _rows;
+    std::shared_ptr<std::vector<roaring::Roaring>> _listed;
+};
+
+// An unscored collection given candidates keeps exactly the matches among 
them. A root that
+// lists its rows lists them within the candidates; any other root is read 
through its scorer,
+// seeking candidate by candidate when they are sparse and intersecting once 
when they are dense.
+TEST(BooleanTruthContractTest, CollectionWithinCandidatesKeepsTheirMatches) {
+    constexpr uint32_t row_count = 4096;
+    TruthRows rows;
+    for (uint32_t doc = 0; doc < row_count; doc += 3) {
+        rows.true_rows->add(doc);
+    }
+    roaring::Roaring sparse;
+    for (uint32_t doc = 0; doc < row_count; doc += 97) {
+        sparse.add(doc);
+    }
+    roaring::Roaring dense;
+    for (uint32_t doc = 0; doc < row_count; doc += 2) {
+        dense.add(doc);
+    }
+    auto collect = [&](const QueryPtr& query, const roaring::Roaring& 
candidates) {
+        QueryExecutionContext context;
+        context.segment_num_rows = row_count;
+        auto collected = std::make_shared<roaring::Roaring>();
+        collect_multi_segment_doc_set(query->weight(false), context, "", 
collected, nullptr,
+                                      /*enable_scoring=*/false, nullptr, 
&candidates);
+        return *collected;
+    };
+
+    for (const auto* candidates : {&sparse, &dense}) {
+        auto listed = std::make_shared<std::vector<roaring::Roaring>>();
+        EXPECT_EQ(collect(std::make_shared<ListingQuery>(rows, listed), 
*candidates),
+                  *rows.true_rows & *candidates);
+        ASSERT_EQ(listed->size(), 1U);
+        EXPECT_EQ(listed->front(), *candidates);
+    }
+
+    // Sparse candidates: the scorer only seeks to them, and never walks the 
rows between.
+    auto work = std::make_shared<ScorerWork>();
+    EXPECT_EQ(collect(std::make_shared<ForwardOnlyQuery>(rows, 1.0F, work), 
sparse),
+              *rows.true_rows & sparse);
+    EXPECT_GT(work->seeks, 0U);
+    EXPECT_LE(work->seeks, sparse.cardinality());
+    EXPECT_EQ(work->advances, 0U);
+
+    // Dense candidates: the scorer is read once and intersected with them.
+    work = std::make_shared<ScorerWork>();
+    EXPECT_EQ(collect(std::make_shared<ForwardOnlyQuery>(rows, 1.0F, work), 
dense),
+              *rows.true_rows & dense);
+    EXPECT_EQ(work->seeks, 0U);
+}
+
 } // namespace
 } // namespace doris::segment_v2::inverted_index::query_v2
diff --git 
a/regression-test/data/search/test_search_outer_filter_candidates.out 
b/regression-test/data/search/test_search_outer_filter_candidates.out
new file mode 100644
index 00000000000..dcdc8370c30
--- /dev/null
+++ b/regression-test/data/search/test_search_outer_filter_candidates.out
@@ -0,0 +1,741 @@
+-- This file is automatically generated. You should know what you did if you 
want to edit this
+-- !idx_lucene_0_rows --
+136
+20
+576
+580
+620
+796
+880
+956
+
+-- !idx_lucene_0_scores --
+136    6.1794
+20     5.7438
+576    6.7006
+580    5.7438
+620    5.7438
+796    6.1794
+880    5.7438
+956    6.1794
+
+-- !idx_lucene_0_cached_filtered --
+8
+
+-- !idx_lucene_0_cached_full --
+92
+
+-- !idx_lucene_0_cached_filtered_again --
+8
+
+-- !idx_standard_0_rows --
+136
+20
+576
+580
+620
+796
+880
+956
+
+-- !idx_standard_0_scores --
+136    6.1794
+20     5.7438
+576    6.7006
+580    5.7438
+620    5.7438
+796    6.1794
+880    5.7438
+956    6.1794
+
+-- !idx_standard_0_cached_filtered --
+8
+
+-- !idx_standard_0_cached_full --
+92
+
+-- !idx_standard_0_cached_filtered_again --
+8
+
+-- !idx_lucene_1_rows --
+20
+294
+354
+454
+580
+614
+620
+880
+
+-- !idx_lucene_1_scores --
+20     2.8998
+294    3.4113
+354    3.4113
+454    2.8909
+580    2.8998
+614    2.8909
+620    2.8998
+880    2.8998
+
+-- !idx_lucene_1_cached_filtered --
+8
+
+-- !idx_lucene_1_cached_full --
+93
+
+-- !idx_lucene_1_cached_filtered_again --
+8
+
+-- !idx_standard_1_rows --
+20
+294
+354
+454
+580
+614
+620
+880
+
+-- !idx_standard_1_scores --
+20     2.8998
+294    3.4113
+354    3.4113
+454    2.8909
+580    2.8998
+614    2.8909
+620    2.8998
+880    2.8998
+
+-- !idx_standard_1_cached_filtered --
+8
+
+-- !idx_standard_1_cached_full --
+93
+
+-- !idx_standard_1_cached_filtered_again --
+8
+
+-- !idx_lucene_2_rows --
+136
+144
+175
+20
+224
+232
+324
+348
+372
+384
+415
+448
+45
+508
+565
+575
+576
+580
+620
+628
+752
+785
+796
+824
+84
+880
+956
+975
+
+-- !idx_lucene_2_scores --
+144    1.5904
+20     2.8065
+324    1.5904
+348    1.5904
+372    1.5904
+384    1.5904
+580    2.8065
+620    2.8065
+84     1.5904
+880    2.8065
+
+-- !idx_lucene_2_cached_filtered --
+28
+
+-- !idx_lucene_2_cached_full --
+334
+
+-- !idx_lucene_2_cached_filtered_again --
+28
+
+-- !idx_standard_2_rows --
+136
+144
+175
+20
+224
+232
+324
+348
+372
+384
+415
+448
+45
+508
+565
+575
+576
+580
+620
+628
+752
+785
+796
+824
+84
+880
+956
+975
+
+-- !idx_standard_2_scores --
+144    1.5904
+20     2.8065
+324    1.5904
+348    1.5904
+372    1.5904
+384    1.5904
+580    2.8065
+620    2.8065
+84     1.5904
+880    2.8065
+
+-- !idx_standard_2_cached_filtered --
+28
+
+-- !idx_standard_2_cached_full --
+334
+
+-- !idx_standard_2_cached_filtered_again --
+28
+
+-- !idx_lucene_3_rows --
+144
+224
+289
+294
+324
+354
+384
+454
+479
+614
+679
+699
+719
+754
+824
+829
+84
+949
+999
+
+-- !idx_lucene_3_scores --
+144    2.9175
+224    2.9175
+289    2.9175
+294    2.9175
+324    2.9175
+354    2.9175
+384    2.9175
+454    2.9175
+479    2.9175
+84     2.9175
+
+-- !idx_lucene_3_cached_filtered --
+19
+
+-- !idx_lucene_3_cached_full --
+200
+
+-- !idx_lucene_3_cached_filtered_again --
+19
+
+-- !idx_standard_3_rows --
+144
+224
+289
+294
+324
+354
+384
+454
+479
+614
+679
+699
+719
+754
+824
+829
+84
+949
+999
+
+-- !idx_standard_3_scores --
+144    2.9175
+224    2.9175
+289    2.9175
+294    2.9175
+324    2.9175
+354    2.9175
+384    2.9175
+454    2.9175
+479    2.9175
+84     2.9175
+
+-- !idx_standard_3_cached_filtered --
+19
+
+-- !idx_standard_3_cached_full --
+200
+
+-- !idx_standard_3_cached_filtered_again --
+19
+
+-- !idx_lucene_4_rows --
+136
+576
+796
+956
+
+-- !idx_lucene_4_scores --
+136    2.8948
+576    3.416
+796    2.8948
+956    2.8948
+
+-- !idx_lucene_4_cached_filtered --
+4
+
+-- !idx_lucene_4_cached_full --
+46
+
+-- !idx_lucene_4_cached_filtered_again --
+4
+
+-- !idx_standard_4_rows --
+136
+576
+796
+956
+
+-- !idx_standard_4_scores --
+136    2.8948
+576    3.416
+796    2.8948
+956    2.8948
+
+-- !idx_standard_4_cached_filtered --
+4
+
+-- !idx_standard_4_cached_full --
+46
+
+-- !idx_standard_4_cached_filtered_again --
+4
+
+-- !idx_lucene_5_rows --
+121
+136
+221
+26
+351
+421
+511
+521
+546
+576
+601
+611
+621
+701
+771
+796
+806
+811
+831
+86
+956
+
+-- !idx_lucene_5_scores --
+121    1.8993
+136    1.8993
+221    1.8993
+26     1.8993
+351    1.8993
+421    1.8993
+511    1.8993
+521    1.8993
+546    1.8993
+86     1.8993
+
+-- !idx_lucene_5_cached_filtered --
+21
+
+-- !idx_lucene_5_cached_full --
+200
+
+-- !idx_lucene_5_cached_filtered_again --
+21
+
+-- !idx_standard_5_rows --
+121
+136
+221
+26
+351
+421
+511
+521
+546
+576
+601
+611
+621
+701
+771
+796
+806
+811
+831
+86
+956
+
+-- !idx_standard_5_scores --
+121    1.8993
+136    1.8993
+221    1.8993
+26     1.8993
+351    1.8993
+421    1.8993
+511    1.8993
+521    1.8993
+546    1.8993
+86     1.8993
+
+-- !idx_standard_5_cached_filtered --
+21
+
+-- !idx_standard_5_cached_full --
+200
+
+-- !idx_standard_5_cached_filtered_again --
+21
+
+-- !key_lucene_0_rows --
+136
+20
+576
+580
+620
+796
+880
+956
+
+-- !key_lucene_0_cached_filtered --
+8
+
+-- !key_lucene_0_cached_full --
+92
+
+-- !key_lucene_0_cached_filtered_again --
+8
+
+-- !key_standard_0_rows --
+136
+20
+576
+580
+620
+796
+880
+956
+
+-- !key_standard_0_cached_filtered --
+8
+
+-- !key_standard_0_cached_full --
+92
+
+-- !key_standard_0_cached_filtered_again --
+8
+
+-- !key_lucene_1_rows --
+20
+294
+354
+454
+580
+614
+620
+880
+
+-- !key_lucene_1_cached_filtered --
+8
+
+-- !key_lucene_1_cached_full --
+93
+
+-- !key_lucene_1_cached_filtered_again --
+8
+
+-- !key_standard_1_rows --
+20
+294
+354
+454
+580
+614
+620
+880
+
+-- !key_standard_1_cached_filtered --
+8
+
+-- !key_standard_1_cached_full --
+93
+
+-- !key_standard_1_cached_filtered_again --
+8
+
+-- !key_lucene_2_rows --
+136
+144
+175
+20
+224
+232
+324
+348
+372
+384
+415
+448
+45
+508
+565
+575
+576
+580
+620
+628
+752
+785
+796
+824
+84
+880
+956
+975
+
+-- !key_lucene_2_cached_filtered --
+28
+
+-- !key_lucene_2_cached_full --
+334
+
+-- !key_lucene_2_cached_filtered_again --
+28
+
+-- !key_standard_2_rows --
+136
+144
+175
+20
+224
+232
+324
+348
+372
+384
+415
+448
+45
+508
+565
+575
+576
+580
+620
+628
+752
+785
+796
+824
+84
+880
+956
+975
+
+-- !key_standard_2_cached_filtered --
+28
+
+-- !key_standard_2_cached_full --
+334
+
+-- !key_standard_2_cached_filtered_again --
+28
+
+-- !key_lucene_3_rows --
+144
+224
+289
+294
+324
+354
+384
+454
+479
+614
+679
+699
+719
+754
+824
+829
+84
+949
+999
+
+-- !key_lucene_3_cached_filtered --
+19
+
+-- !key_lucene_3_cached_full --
+200
+
+-- !key_lucene_3_cached_filtered_again --
+19
+
+-- !key_standard_3_rows --
+144
+224
+289
+294
+324
+354
+384
+454
+479
+614
+679
+699
+719
+754
+824
+829
+84
+949
+999
+
+-- !key_standard_3_cached_filtered --
+19
+
+-- !key_standard_3_cached_full --
+200
+
+-- !key_standard_3_cached_filtered_again --
+19
+
+-- !key_lucene_4_rows --
+136
+576
+796
+956
+
+-- !key_lucene_4_cached_filtered --
+4
+
+-- !key_lucene_4_cached_full --
+46
+
+-- !key_lucene_4_cached_filtered_again --
+4
+
+-- !key_standard_4_rows --
+136
+576
+796
+956
+
+-- !key_standard_4_cached_filtered --
+4
+
+-- !key_standard_4_cached_full --
+46
+
+-- !key_standard_4_cached_filtered_again --
+4
+
+-- !key_lucene_5_rows --
+121
+136
+221
+26
+351
+421
+511
+521
+546
+576
+601
+611
+621
+701
+771
+796
+806
+811
+831
+86
+956
+
+-- !key_lucene_5_cached_filtered --
+21
+
+-- !key_lucene_5_cached_full --
+200
+
+-- !key_lucene_5_cached_filtered_again --
+21
+
+-- !key_standard_5_rows --
+121
+136
+221
+26
+351
+421
+511
+521
+546
+576
+601
+611
+621
+701
+771
+796
+806
+811
+831
+86
+956
+
+-- !key_standard_5_cached_filtered --
+21
+
+-- !key_standard_5_cached_full --
+200
+
+-- !key_standard_5_cached_filtered_again --
+21
+
diff --git 
a/regression-test/suites/search/test_search_outer_filter_candidates.groovy 
b/regression-test/suites/search/test_search_outer_filter_candidates.groovy
new file mode 100644
index 00000000000..f2af20b714f
--- /dev/null
+++ b/regression-test/suites/search/test_search_outer_filter_candidates.groovy
@@ -0,0 +1,107 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// A SEARCH that is a whole conjunct starts from the rows the scan's other 
filters kept, read
+// through an inverted index or a sort key. The expected rows and scores in 
the .out file are
+// those of the SEARCH over the whole segment, filtered afterwards; with the 
query cache on, a
+// partial result must not reach a later query without the filter.
+suite("test_search_outer_filter_candidates") {
+    def tables = [
+        idx: "test_search_outer_filter_idx",
+        key: "test_search_outer_filter_key",
+    ]
+    def textIndex = 'USING INVERTED PROPERTIES("parser" = "unicode", 
"lower_case" = "true", "support_phrase" = "true")'
+    tables.each { kind, table ->
+        sql "DROP TABLE IF EXISTS ${table}"
+        def columns = kind == "idx" ? "id INT, account INT" : "account INT NOT 
NULL, id INT"
+        def accountIndex = kind == "idx" ? ", INDEX idx_account (account) 
USING INVERTED" : ""
+        sql """
+            CREATE TABLE ${table} (
+                ${columns},
+                title TEXT,
+                skill TEXT,
+                place TEXT,
+                INDEX idx_title (title) ${textIndex},
+                INDEX idx_skill (skill) ${textIndex},
+                INDEX idx_place (place) ${textIndex}
+                ${accountIndex}
+            ) ENGINE=OLAP
+            DUPLICATE KEY(${kind == "idx" ? "id" : "account, id"})
+            DISTRIBUTED BY HASH(id) BUCKETS 1
+            PROPERTIES ("replication_allocation" = "tag.location.default: 1")
+        """
+        // Account 7 holds about a tenth of the rows, spread by a hash so its 
titles, skills and
+        // places vary like the others'; NULL titles on every 13th row.
+        sql """
+            INSERT INTO ${table} (id, account, title, skill, place)
+            SELECT number,
+                   if(abs(murmur_hash3_32(concat('account-', number))) % 10 = 
0, 7, number % 50 + 100),
+                   if(number % 13 = 0, NULL, concat_ws(' ',
+                       element_at(['', 'senior', 'lead'], number % 3 + 1),
+                       element_at(['software', 'cloud', 'sales', 'data', 
'ai'], number % 5 + 1),
+                       element_at(['engineer', 'manager', 'developer', 
'analyst'], number % 4 + 1))),
+                   element_at(['python sql', 'docker', 'team leadership', 
'excel', 'project management'],
+                              number % 5 + 1),
+                   element_at(['Manila PH', 'Cebu Philippines', 'Berlin 
Germany', 'Pune India'],
+                              number % 4 + 1)
+            FROM numbers("number" = "1000")
+        """
+    }
+    sql "sync"
+    sql "set enable_common_expr_pushdown = true"
+
+    def dsls = [
+        // AND of OR groups, as candidate search writes them
+        '(skill:ANY(python docker excel) OR skill:"team leadership") AND ' +
+            '(title:"software engineer" OR title:"cloud engineer" OR 
title:"data analyst") AND ' +
+            '(place:ANY(ph pilipinas) OR place:"cebu philippines")',
+        'title:"software engineer" OR title:"ai developer"',
+        '(title:ANY(engineer developer) OR skill:sql) AND NOT place:berlin',
+        'skill:"project management" AND NOT title:"sales manager"',
+        // a phrase root, which lists its rows itself
+        'title:"cloud engineer"',
+        // a term root, read through its scorer
+        'skill:docker',
+    ]
+    def modes = [lucene: '{"mode":"lucene"}', standard: '{"mode":"standard"}']
+
+    tables.each { kind, table ->
+        dsls.eachWithIndex { dsl, i ->
+            modes.each { modeName, mode ->
+                def search = "search('${dsl}', '${mode}')"
+                def tag = "${kind}_${modeName}_${i}"
+                sql "set enable_inverted_index_query_cache = false"
+                "order_qt_${tag}_rows"("SELECT id FROM ${table} WHERE account 
= 7 AND ${search}")
+                if (kind == "idx") {
+                    "order_qt_${tag}_scores"("""
+                        SELECT id, ROUND(score(), 4) AS s FROM ${table} WHERE 
account = 7 AND ${search}
+                        ORDER BY score() DESC, id LIMIT 10""")
+                }
+
+                // With the cache on, the filtered query first: the query 
without the filter
+                // must still count every row of the SEARCH.
+                sql "set enable_inverted_index_query_cache = true"
+                "qt_${tag}_cached_filtered"("SELECT count(*) FROM ${table} 
WHERE account = 7 AND ${search}")
+                "qt_${tag}_cached_full"("SELECT count(*) FROM ${table} WHERE 
${search}")
+                "qt_${tag}_cached_filtered_again"("SELECT count(*) FROM 
${table} WHERE account = 7 AND ${search}")
+            }
+        }
+    }
+    sql "set enable_inverted_index_query_cache = false"
+
+    tables.each { kind, table -> sql "DROP TABLE IF EXISTS ${table}" }
+}


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to