This is an automated email from the ASF dual-hosted git repository.

mrhhsg pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
     new d8e9d1ac448 [refactor](be) Simplify runtime filter merger ownership 
(#70661)
d8e9d1ac448 is described below

commit d8e9d1ac44821b15843ec02aeef8f13644a8f9e8
Author: Jerry Hu <[email protected]>
AuthorDate: Thu Nov 19 11:50:50 2026 +0800

    [refactor](be) Simplify runtime filter merger ownership (#70661)
    
    ### Background
    
    Runtime filter producers can publish to direct local consumers and to
    local or global mergers. A merger accumulates producer filters before
    the merged filter is delivered locally or sent to remote consumers.
    
    ### What problem does this PR solve?
    
    Issue Number: None
    Related PR: #68684 (established private-copy behavior for shared inputs)
    PR Type: Refactor
    
    Problem Summary: The merger previously accepted an exclusive-ownership
    hint from its callers to avoid copying some mutable inputs. That
    contract depends on producer routing and RPC-input lifetime assumptions
    spread across several layers. This refactor makes the merger accumulate
    in a private copy and makes IN-to-Bloom conversion copy its input Bloom
    before inserting IN values. Filter results and delivery routing remain
    unchanged; formerly exclusive paths now perform an additional copy.
    
    ### Details
    
    - **Moves vs. logic changes:** Remove the ownership hints, producer-side
    ownership inference, and direct-adoption branches; retain the existing
    publication, readiness, and RPC routing.
    - **Behavior change:** Clone the first merger input even when a caller
    previously considered it exclusive, and clone a transitional Bloom input
    before merging IN values into it.
    - **Compatibility:** Internal BE C++ API only; no SQL, RPC wire, or
    persisted-format changes.
    - **Performance impact:** Formerly exclusive paths incur an additional
    deep copy under the existing merger lock. Production performance and
    peak-memory impact have not been benchmarked.
    - **Rollback:** Revert this refactor if the added copy cost is
    unacceptable.
    
    ### Testing
    
    - Local ASAN BE/FE build, including FE checkstyle: `BUILD_TYPE=ASAN
    ./build.sh --be --fe -j 16` passed.
    - Local BE unit tests: the six focused ownership/clone cases passed; the
    five changed ownership assertions failed against the old implementation
    as expected. `./run-be-ut.sh --run -j 16
    --filter='*RuntimeFilter*:*BloomFilter*:*MinMax*:*SyncSize*'` passed
    419/419 tests.
    - Local single-BE SQL regression via `run-regression-test.sh`: `rfv2`,
    `set_operator_in_filter`, and `rf_bucket_pruning` each passed (one suite
    each).
    - `build-support/clang-format.sh`, `build-support/check-format.sh`,
    `build-support/check-build-hygiene.sh`, and `git diff --check` passed.
    The changed-line `build-support/run-clang-tidy.sh` check passed with the
    Clang resource directory configured.
    - Not run: multi-BE RPC end-to-end testing, mixed-version testing, or a
    production-performance/peak-memory A/B benchmark.
    
    ### Release note
    
    None
    
    ### Check List (For Author)
    
    - Behavior changed: Yes (see Details)
    - Does this need documentation: No
    - Pre-submission status: Partially verified: multi-BE behavior and
    production performance remain unmeasured.
    
    ### AI Disclosure
    
    - Tool: Codex
    - Generated: Code, unit tests, and this description
    - Human review: Not yet performed
---
 be/src/exec/runtime_filter/runtime_filter_merger.h |  50 ++++----
 be/src/exec/runtime_filter/runtime_filter_mgr.cpp  |   6 +-
 .../runtime_filter/runtime_filter_producer.cpp     |  55 ++-------
 .../exec/runtime_filter/runtime_filter_wrapper.cpp |  14 +--
 .../exec/runtime_filter/runtime_filter_wrapper.h   |   6 +-
 .../runtime_filter/runtime_filter_merger_test.cpp  | 132 +++++++++++++++++++--
 .../runtime_filter_producer_test.cpp               |  77 ++++++++----
 .../runtime_filter/runtime_filter_wrapper_test.cpp |  47 ++++++--
 8 files changed, 255 insertions(+), 132 deletions(-)

diff --git a/be/src/exec/runtime_filter/runtime_filter_merger.h 
b/be/src/exec/runtime_filter/runtime_filter_merger.h
index 3e0b465b1f8..b6edef83f1d 100644
--- a/be/src/exec/runtime_filter/runtime_filter_merger.h
+++ b/be/src/exec/runtime_filter/runtime_filter_merger.h
@@ -56,37 +56,41 @@ public:
                 _received_rf_size_num, _received_sum_size);
     }
 
-    // If input is a disabled predicate, the final result is a disabled 
predicate.
+    // Keep a private copy of the first input: later merges must not change a 
producer wrapper
+    // which a local consumer may already be probing. A disabled input 
disables the result.
     // Returns true only for the call that makes the merger ready.
-    //
-    // `other_wrapper_exclusively_owned` must be true only when the caller 
guarantees `other`'s
-    // wrapper has no other reader and will not be read again after this call 
(e.g. a producer
-    // whose wrapper is never signaled to a plain local consumer, or an 
RPC-only filter built
-    // solely to carry one merge request). The merger then takes the wrapper 
over directly
-    // instead of paying for a deep copy. It must stay false whenever 
`other`'s wrapper may still
-    // be used by consumers in local RF mgr of the same instance, or is shared 
by all producers of
-    // a broadcast join with a shared hash table: the merger must never write 
such a wrapper, so
-    // it only merges into its own copy.
-    Status merge_from(const RuntimeFilter* other, bool* ready,
-                      bool other_wrapper_exclusively_owned = false) {
+    Status merge_from(const RuntimeFilter* other, bool* ready) {
         std::unique_lock<std::recursive_mutex> l(_rmtx);
-        _received_producer_num++;
-        if (_expected_producer_num < _received_producer_num) {
+        *ready = false;
+        if (_expected_producer_num <= _received_producer_num) {
             return Status::InternalError(
                     "runtime filter merger input product more than expected, 
{}", debug_string());
         }
+        Status status;
+        if (_wrapper->get_state() == RuntimeFilterWrapper::State::UNINITED) {
+            // For example, consider a join whose probe side has two scan 
targets:
+            //   SELECT p.k FROM build_table b JOIN
+            //     (SELECT k FROM local_probe UNION ALL SELECT k FROM 
merged_probe) p ON b.k = p.k;
+            // Depending on the physical plan, the same RF can be routed as 
follows:
+            //   producer 1 RF {5} -----> Scan(local_probe)  [direct, no merge]
+            //           |-- copy --> merger <-- producer 2 RF {6}
+            //                         |
+            //                         +----> Scan(merged_probe) [merged: {5, 
6}]
+            // The merger must clone producer 1's wrapper: adopting it would 
let the
+            // later merge change the direct scan's filter from {5} to {5, 6}.
+            status = other->_wrapper->clone(&_wrapper);
+        } else {
+            status = _wrapper->merge(other->_wrapper.get());
+        }
+        RETURN_IF_ERROR(status);
+        // A failed copy or merge must not count as an arrival: otherwise 
later inputs could
+        // publish a filter that does not contain the failed producer's values.
+        ++_received_producer_num;
         *ready = _received_producer_num == _expected_producer_num;
-        if (_received_producer_num == _expected_producer_num) {
+        if (*ready) {
             _rf_state = State::READY;
         }
-        if (_wrapper->get_state() == RuntimeFilterWrapper::State::UNINITED) {
-            if (other_wrapper_exclusively_owned) {
-                _wrapper = other->_wrapper;
-                return Status::OK();
-            }
-            return other->_wrapper->clone(&_wrapper);
-        }
-        return _wrapper->merge(other->_wrapper.get(), 
other_wrapper_exclusively_owned);
+        return Status::OK();
     }
 
     // Only raise the expected producer count. RuntimeFilterMgr may compute the
diff --git a/be/src/exec/runtime_filter/runtime_filter_mgr.cpp 
b/be/src/exec/runtime_filter/runtime_filter_mgr.cpp
index 468e8851720..29912f5c441 100644
--- a/be/src/exec/runtime_filter/runtime_filter_mgr.cpp
+++ b/be/src/exec/runtime_filter/runtime_filter_mgr.cpp
@@ -589,11 +589,7 @@ Status 
RuntimeFilterMergeControllerEntity::merge(std::shared_ptr<QueryContext> q
 
         RETURN_IF_ERROR(tmp_filter->assign(*request, attach_data));
 
-        // `tmp_filter` is created solely to carry this one RPC's filter into 
the merger and is
-        // discarded right after; nothing else reads its wrapper, so the 
merger may take it over
-        // without a copy.
-        RETURN_IF_ERROR(cnt_val.merger->merge_from(tmp_filter.get(), &is_ready,
-                                                   
/*other_wrapper_exclusively_owned=*/true));
+        RETURN_IF_ERROR(cnt_val.merger->merge_from(tmp_filter.get(), 
&is_ready));
 
         cnt_val.arrive_id.insert(UniqueId(request->fragment_instance_id()));
     }
diff --git a/be/src/exec/runtime_filter/runtime_filter_producer.cpp 
b/be/src/exec/runtime_filter/runtime_filter_producer.cpp
index 2a9839111ee..25deb4ab45b 100644
--- a/be/src/exec/runtime_filter/runtime_filter_producer.cpp
+++ b/be/src/exec/runtime_filter/runtime_filter_producer.cpp
@@ -27,17 +27,6 @@
 #include "util/brpc_closure.h"
 
 namespace doris {
-namespace {
-// Nothing other than the merger can read the wrapper `merge_from` is about to 
receive: either no
-// plain local consumer of `state` exists at all 
(`_send_to_local_targets(state, this, false)` is
-// then a no-op), or `merger` expects only this one producer, so `merge_from` 
becomes ready on
-// this very call and never merges another producer into the wrapper it adopts 
here, even though
-// a plain local consumer also reads it.
-bool wrapper_has_no_other_reader(RuntimeState* state, int filter_id, 
RuntimeFilterMerger* merger) {
-    return 
state->local_runtime_filter_mgr()->get_consume_filters(filter_id).empty() ||
-           merger->get_expected_producer_num() == 1;
-}
-} // namespace
 
 Status RuntimeFilterProducer::_send_to_remote_targets(RuntimeState* state,
                                                       RuntimeFilter* 
merger_filter) {
@@ -57,21 +46,14 @@ Status 
RuntimeFilterProducer::_send_to_local_targets(RuntimeState* state, Runtim
     return Status::OK();
 };
 
-// Pre-existing complexity debt from the target/broadcast/remote branching 
below, already over
-// threshold before this change; the ownership refinement above is factored 
out to keep the
-// increase small, but not addressed further here to keep this diff focused.
+// Pre-existing complexity debt from the target/broadcast/remote branching 
below is not addressed
+// here to keep this change focused.
 // NOLINTNEXTLINE(readability-function-cognitive-complexity)
 Status RuntimeFilterProducer::publish(RuntimeState* state, bool 
build_hash_table) {
     std::unique_lock<std::recursive_mutex> l(_rmtx);
     _check_state({State::READY_TO_PUBLISH});
 
-    // `allow_relaxed_ownership` is only meaningful when 
`other_wrapper_exclusively_owned` is
-    // false: it lets this call refine that decision using the local-merge 
context it fetches
-    // below, instead of the caller guessing it without the context. It must 
stay false for the
-    // remote-target call site, whose `true` already holds unconditionally for 
a different
-    // reason (see its call below), and for the broadcast case, whose 
producers alias one shared
-    // wrapper (see 
`RuntimeFilterProducerTest.publish_mixed_targets_shared_wrapper`).
-    auto do_merge = [&](bool other_wrapper_exclusively_owned, bool 
allow_relaxed_ownership) {
+    auto do_merge = [&]() {
         if (!_need_do_merge(state)) {
             // when global consumer not exist, send_to_local_targets will do 
nothing, so merge rf is useless
             return Status::OK();
@@ -83,12 +65,8 @@ Status RuntimeFilterProducer::publish(RuntimeState* state, 
bool build_hash_table
             // Filter was removed during a recursive CTE stage reset; this 
producer is stale.
             return Status::OK();
         }
-        if (!other_wrapper_exclusively_owned && allow_relaxed_ownership) {
-            other_wrapper_exclusively_owned = wrapper_has_no_other_reader(
-                    state, _wrapper->filter_id(), context->merger.get());
-        }
         bool ready = false;
-        RETURN_IF_ERROR(context->merger->merge_from(this, &ready, 
other_wrapper_exclusively_owned));
+        RETURN_IF_ERROR(context->merger->merge_from(this, &ready));
         if (ready) {
             if (_has_remote_target) {
                 RETURN_IF_ERROR(_send_to_remote_targets(state, 
context->merger.get()));
@@ -102,30 +80,15 @@ Status RuntimeFilterProducer::publish(RuntimeState* state, 
bool build_hash_table
     if (!_has_remote_target) {
         // A runtime filter may have multiple targets and some of those are 
local-merge RF and others are not.
         // So for all runtime filters' producers, `publish` should notify all 
consumers in global RF mgr which manages local-merge RF and local RF mgr which 
manages others.
-        // The merger never writes this wrapper while a plain local consumer 
may still read it
-        // (see `RuntimeFilterMerger::merge_from`), so by default the 
consumers in local RF mgr
-        // can use it right away while the merge of the other producers goes 
on. `do_merge` may
-        // still find no such consumer exists (or this is the only producer) 
and adopt the
-        // wrapper directly. Broadcast producers alias one shared wrapper 
across instances (see
-        // `publish_mixed_targets_shared_wrapper`), so relaxing ownership is 
unsound for them;
-        // `build_bf_by_runtime_size` is excluded too, to keep this call's 
precondition the same
-        // simple "nothing else can reach this wrapper" rule as the 
remote-target call below,
-        // without re-deriving it for a filter whose size-sync step this 
branch never exercises.
-        RETURN_IF_ERROR(do_merge(/*other_wrapper_exclusively_owned=*/false,
-                                 
/*allow_relaxed_ownership=*/!_is_broadcast_join &&
-                                         
!_wrapper->build_bf_by_runtime_size()));
+        // The merger copies this wrapper before later merges, so plain local 
consumers can read
+        // the producer wrapper independently, including with mixed local and 
merge targets.
+        RETURN_IF_ERROR(do_merge());
         RETURN_IF_ERROR(_send_to_local_targets(state, this, false));
     } else if (build_hash_table) {
         if (_is_broadcast_join) {
             RETURN_IF_ERROR(_send_to_remote_targets(state, this));
         } else {
-            // This path never hands `_wrapper` to a plain local consumer 
(that only happens in
-            // the `!_has_remote_target` branch above), and the shared-wrapper 
broadcast-join
-            // case is the other arm of this `if`, so no other producer 
aliases it either. The
-            // only remaining reference after this call is the one 
`_wrapper.reset()` below
-            // drops, so the merger may take `_wrapper` over directly instead 
of cloning it.
-            RETURN_IF_ERROR(do_merge(/*other_wrapper_exclusively_owned=*/true,
-                                     /*allow_relaxed_ownership=*/false));
+            RETURN_IF_ERROR(do_merge());
         }
     } else {
         if (!_is_broadcast_join) {
@@ -135,7 +98,7 @@ Status RuntimeFilterProducer::publish(RuntimeState* state, 
bool build_hash_table
         }
     }
 
-    // wrapper may moved to rf merger, release wrapper here to make sure 
thread safe
+    // Release the producer reference after consumers and the merger hold 
their own references.
     _wrapper.reset();
     set_state(State::PUBLISHED);
     return Status::OK();
diff --git a/be/src/exec/runtime_filter/runtime_filter_wrapper.cpp 
b/be/src/exec/runtime_filter/runtime_filter_wrapper.cpp
index 9125b250c32..f47cb552b13 100644
--- a/be/src/exec/runtime_filter/runtime_filter_wrapper.cpp
+++ b/be/src/exec/runtime_filter/runtime_filter_wrapper.cpp
@@ -128,8 +128,7 @@ bool RuntimeFilterWrapper::build_bf_by_runtime_size() const 
{
 // Pre-existing size/complexity debt from the per-filter-type switch below, 
already over
 // threshold before this change; not addressed here to keep this diff focused.
 // 
NOLINTNEXTLINE(readability-function-size,readability-function-cognitive-complexity)
-Status RuntimeFilterWrapper::merge(const RuntimeFilterWrapper* other,
-                                   bool other_exclusively_owned) {
+Status RuntimeFilterWrapper::merge(const RuntimeFilterWrapper* other) {
     DORIS_CHECK(!_bucket_prune_hashes_started.load());
     if (_state == State::DISABLED) {
         return Status::OK();
@@ -193,14 +192,9 @@ Status RuntimeFilterWrapper::merge(const 
RuntimeFilterWrapper* other,
                     
RETURN_IF_ERROR(_bloom_filter_func->init_with_fixed_length(0));
                     RETURN_IF_ERROR(_change_to_bloom_filter());
                 }
-            } else if (other_exclusively_owned) {
-                // `other` has no other reader, so its bf can be taken over 
directly instead of
-                // cloned before inserting the hybrid set data into it.
-                _bloom_filter_func = other->_bloom_filter_func;
-                RETURN_IF_ERROR(_change_to_bloom_filter());
             } else {
-                // case1&case2: use a copy of input bf and insert hybrid set 
data into it. `other`
-                // may still be used by its own consumers, so it must not be 
written.
+                // Use a private copy before adding this IN set; the input 
Bloom may still be
+                // read by its own consumers.
                 
RETURN_IF_ERROR(other->_bloom_filter_func->clone(&_bloom_filter_func, true));
                 RETURN_IF_ERROR(_change_to_bloom_filter());
             }
@@ -237,7 +231,7 @@ Status 
RuntimeFilterWrapper::clone(std::shared_ptr<RuntimeFilterWrapper>* res) c
     if (_bloom_filter_func) {
         // An IN_OR_BLOOM filter which is still an IN filter never reads its 
bloom filter: a
         // later merge which changes the copy to a bloom filter either 
re-initializes the bloom
-        // filter or takes a copy of the input one first. So do not copy the 
unused bloom filter.
+        // filter or clones the input one first. So do not copy the unused 
bloom filter.
         RETURN_IF_ERROR(_bloom_filter_func->clone(
                 &cloned->_bloom_filter_func, get_real_type() == 
RuntimeFilterType::BLOOM_FILTER));
     }
diff --git a/be/src/exec/runtime_filter/runtime_filter_wrapper.h 
b/be/src/exec/runtime_filter/runtime_filter_wrapper.h
index 8aa303733d0..8faeba32b29 100644
--- a/be/src/exec/runtime_filter/runtime_filter_wrapper.h
+++ b/be/src/exec/runtime_filter/runtime_filter_wrapper.h
@@ -57,10 +57,8 @@ public:
 
     Status init(const size_t runtime_size);
     Status insert(const ColumnPtr& column, size_t start);
-    // `other_exclusively_owned` must be true only when `other` has no other 
reader and will not
-    // be read again after this call: a transitional BLOOM directory taken 
from it is then moved
-    // in directly instead of cloned. See `RuntimeFilterMerger::merge_from`.
-    Status merge(const RuntimeFilterWrapper* other, bool 
other_exclusively_owned = false);
+    // Merge into this wrapper without modifying the input.
+    Status merge(const RuntimeFilterWrapper* other);
     // Deep copy of the filter. The result shares no mutable data with this 
wrapper, so the
     // merger can keep merging into the copy while this one is already used by 
consumers.
     Status clone(std::shared_ptr<RuntimeFilterWrapper>* res) const;
diff --git a/be/test/exec/runtime_filter/runtime_filter_merger_test.cpp 
b/be/test/exec/runtime_filter/runtime_filter_merger_test.cpp
index 5823d39797b..2fe78d5a45c 100644
--- a/be/test/exec/runtime_filter/runtime_filter_merger_test.cpp
+++ b/be/test/exec/runtime_filter/runtime_filter_merger_test.cpp
@@ -24,6 +24,8 @@
 #include "exec/runtime_filter/runtime_filter_producer.h"
 #include "exec/runtime_filter/runtime_filter_test_utils.h"
 #include "exprs/hybrid_set.h"
+#include "runtime/memory/mem_tracker_limiter.h"
+#include "runtime/thread_context.h"
 #include "testutil/column_helper.h"
 
 namespace doris {
@@ -162,10 +164,8 @@ TEST_F(RuntimeFilterMergerTest, 
merge_from_never_writes_producer_wrapper) {
               hashes);
 }
 
-// A caller who guarantees `other`'s wrapper has no other reader (e.g. an 
RPC-only filter built
-// solely for this merge, see `RuntimeFilterMergeControllerEntity::merge`) may 
let the merger take
-// it over directly instead of paying for a deep copy.
-TEST_F(RuntimeFilterMergerTest, merge_from_exclusively_owned_takes_ownership) {
+// A deserialized RPC input is short lived; the merger must keep a private 
copy after it dies.
+TEST_F(RuntimeFilterMergerTest, merge_from_rpc_input_uses_private_copy) {
     std::shared_ptr<RuntimeFilterMerger> merger;
     auto desc = TRuntimeFilterDescBuilder().build();
     
FAIL_IF_ERROR_OR_CATCH_EXCEPTION(RuntimeFilterMerger::create(_query_ctx.get(), 
&desc, &merger));
@@ -178,14 +178,26 @@ TEST_F(RuntimeFilterMergerTest, 
merge_from_exclusively_owned_takes_ownership) {
     FAIL_IF_ERROR_OR_CATCH_EXCEPTION(
             producer->insert(ColumnHelper::create_column<DataTypeInt32>({5}), 
0));
     
producer->set_wrapper_state_and_ready_to_publish(RuntimeFilterWrapper::State::READY);
-    auto wrapper = producer->wrapper();
+    PMergeFilterRequest request;
+    void* data = nullptr;
+    int len = 0;
+    FAIL_IF_ERROR_OR_CATCH_EXCEPTION(producer->serialize(&request, &data, 
&len));
+    butil::IOBuf buf;
+    buf.append(data, len);
+    butil::IOBufAsZeroCopyInputStream stream(buf);
+    std::shared_ptr<RuntimeFilterProducer> rpc_input;
+    FAIL_IF_ERROR_OR_CATCH_EXCEPTION(
+            RuntimeFilterProducer::create(_query_ctx.get(), &desc, 
&rpc_input));
+    FAIL_IF_ERROR_OR_CATCH_EXCEPTION(rpc_input->assign(request, &stream));
+    auto wrapper = rpc_input->wrapper();
 
     bool ready = false;
-    FAIL_IF_ERROR_OR_CATCH_EXCEPTION(
-            merger->merge_from(producer.get(), &ready, 
/*other_wrapper_exclusively_owned=*/true));
+    FAIL_IF_ERROR_OR_CATCH_EXCEPTION(merger->merge_from(rpc_input.get(), 
&ready));
     ASSERT_FALSE(ready);
-    // No clone: the merger took over the exclusively owned wrapper directly.
-    ASSERT_EQ(merger->_wrapper, wrapper);
+    ASSERT_NE(merger->_wrapper, wrapper);
+    ASSERT_NE(merger->_wrapper->hybrid_set(), wrapper->hybrid_set());
+    auto merged_wrapper = merger->_wrapper;
+    rpc_input.reset();
 
     std::shared_ptr<RuntimeFilterProducer> producer2;
     FAIL_IF_ERROR_OR_CATCH_EXCEPTION(
@@ -194,10 +206,110 @@ TEST_F(RuntimeFilterMergerTest, 
merge_from_exclusively_owned_takes_ownership) {
     FAIL_IF_ERROR_OR_CATCH_EXCEPTION(
             producer2->insert(ColumnHelper::create_column<DataTypeInt32>({6}), 
0));
     
producer2->set_wrapper_state_and_ready_to_publish(RuntimeFilterWrapper::State::READY);
+    request.Clear();
+    FAIL_IF_ERROR_OR_CATCH_EXCEPTION(producer2->serialize(&request, &data, 
&len));
+    butil::IOBuf buf2;
+    buf2.append(data, len);
+    butil::IOBufAsZeroCopyInputStream stream2(buf2);
+    std::shared_ptr<RuntimeFilterProducer> rpc_input2;
     FAIL_IF_ERROR_OR_CATCH_EXCEPTION(
-            merger->merge_from(producer2.get(), &ready, 
/*other_wrapper_exclusively_owned=*/true));
+            RuntimeFilterProducer::create(_query_ctx.get(), &desc, 
&rpc_input2));
+    FAIL_IF_ERROR_OR_CATCH_EXCEPTION(rpc_input2->assign(request, &stream2));
+    auto wrapper2 = rpc_input2->wrapper();
+    FAIL_IF_ERROR_OR_CATCH_EXCEPTION(merger->merge_from(rpc_input2.get(), 
&ready));
     ASSERT_TRUE(ready);
+    ASSERT_EQ(merger->_wrapper, merged_wrapper);
     ASSERT_EQ(merger->_wrapper->hybrid_set()->size(), 2);
+    int32_t five = 5;
+    int32_t six = 6;
+    ASSERT_TRUE(merged_wrapper->hybrid_set()->find(&five));
+    ASSERT_TRUE(merged_wrapper->hybrid_set()->find(&six));
+    ASSERT_EQ(wrapper->hybrid_set()->size(), 1);
+    ASSERT_TRUE(wrapper->hybrid_set()->find(&five));
+    ASSERT_FALSE(wrapper->hybrid_set()->find(&six));
+    ASSERT_EQ(wrapper2->hybrid_set()->size(), 1);
+    ASSERT_FALSE(wrapper2->hybrid_set()->find(&five));
+}
+
+TEST_F(RuntimeFilterMergerTest, 
failed_rpc_clone_does_not_publish_partial_filter) {
+    auto options = _query_options;
+    options.__set_runtime_filter_max_in_num(1024);
+    options.__set_ignore_runtime_filter_error(true);
+    options.__set_enable_reserve_memory(false);
+    TNetworkAddress fe_address;
+    fe_address.hostname = LOCALHOST;
+    fe_address.port = DUMMY_PORT;
+    auto query_ctx = QueryContext::create(TUniqueId(), ExecEnv::GetInstance(), 
options, fe_address,
+                                          true, fe_address, 
QuerySource::INTERNAL_FRONTEND);
+    ASSERT_TRUE(query_ctx->ignore_runtime_filter_error());
+
+    auto desc = TRuntimeFilterDescBuilder()
+                        .set_type(TRuntimeFilterType::IN_OR_BLOOM)
+                        .set_build_bf_by_runtime_size(false)
+                        .set_bloom_filter_size_bytes(4096)
+                        .set_mode(false)
+                        .build();
+    std::shared_ptr<RuntimeFilterMerger> merger;
+    
FAIL_IF_ERROR_OR_CATCH_EXCEPTION(RuntimeFilterMerger::create(query_ctx.get(), 
&desc, &merger));
+    merger->increase_expected_producer_num(2);
+
+    std::shared_ptr<RuntimeFilterProducer> bloom_source;
+    FAIL_IF_ERROR_OR_CATCH_EXCEPTION(
+            RuntimeFilterProducer::create(query_ctx.get(), &desc, 
&bloom_source));
+    FAIL_IF_ERROR_OR_CATCH_EXCEPTION(bloom_source->init(2000));
+    FAIL_IF_ERROR_OR_CATCH_EXCEPTION(
+            
bloom_source->insert(ColumnHelper::create_column<DataTypeInt32>({5}), 0));
+    
bloom_source->set_wrapper_state_and_ready_to_publish(RuntimeFilterWrapper::State::READY);
+    ASSERT_EQ(bloom_source->wrapper()->get_real_type(), 
RuntimeFilterType::BLOOM_FILTER);
+
+    PMergeFilterRequest request;
+    void* data = nullptr;
+    int len = 0;
+    FAIL_IF_ERROR_OR_CATCH_EXCEPTION(bloom_source->serialize(&request, &data, 
&len));
+    butil::IOBuf buf;
+    buf.append(data, len);
+    butil::IOBufAsZeroCopyInputStream stream(buf);
+    std::shared_ptr<RuntimeFilterProducer> rpc_bloom;
+    FAIL_IF_ERROR_OR_CATCH_EXCEPTION(
+            RuntimeFilterProducer::create(query_ctx.get(), &desc, &rpc_bloom));
+    FAIL_IF_ERROR_OR_CATCH_EXCEPTION(rpc_bloom->assign(request, &stream));
+    ASSERT_EQ(rpc_bloom->wrapper()->get_real_type(), 
RuntimeFilterType::BLOOM_FILTER);
+
+    bool ready = false;
+    auto limited_tracker = 
MemTrackerLimiter::create_shared(MemTrackerLimiter::Type::OTHER,
+                                                            
"FailedRuntimeFilterClone", 2048);
+    {
+        SwitchThreadMemTrackerLimiter switch_tracker(limited_tracker);
+        auto status = merger->merge_from(rpc_bloom.get(), &ready);
+        ASSERT_EQ(status.code(), ErrorCode::MEM_LIMIT_EXCEEDED) << 
status.to_string();
+    }
+    rpc_bloom.reset();
+    ASSERT_FALSE(ready);
+    ASSERT_FALSE(merger->ready());
+    ASSERT_EQ(merger->_wrapper->get_state(), 
RuntimeFilterWrapper::State::UNINITED);
+
+    std::shared_ptr<RuntimeFilterProducer> in_source;
+    FAIL_IF_ERROR_OR_CATCH_EXCEPTION(
+            RuntimeFilterProducer::create(query_ctx.get(), &desc, &in_source));
+    FAIL_IF_ERROR_OR_CATCH_EXCEPTION(in_source->init(1));
+    FAIL_IF_ERROR_OR_CATCH_EXCEPTION(
+            in_source->insert(ColumnHelper::create_column<DataTypeInt32>({6}), 
0));
+    
in_source->set_wrapper_state_and_ready_to_publish(RuntimeFilterWrapper::State::READY);
+    request.Clear();
+    FAIL_IF_ERROR_OR_CATCH_EXCEPTION(in_source->serialize(&request, &data, 
&len));
+    butil::IOBuf empty_buf;
+    butil::IOBufAsZeroCopyInputStream empty_stream(empty_buf);
+    std::shared_ptr<RuntimeFilterProducer> rpc_in;
+    FAIL_IF_ERROR_OR_CATCH_EXCEPTION(
+            RuntimeFilterProducer::create(query_ctx.get(), &desc, &rpc_in));
+    FAIL_IF_ERROR_OR_CATCH_EXCEPTION(rpc_in->assign(request, &empty_stream));
+    ASSERT_EQ(rpc_in->wrapper()->get_real_type(), 
RuntimeFilterType::IN_FILTER);
+
+    FAIL_IF_ERROR_OR_CATCH_EXCEPTION(merger->merge_from(rpc_in.get(), &ready));
+    ASSERT_FALSE(ready);
+    ASSERT_FALSE(merger->ready());
+    ASSERT_EQ(merger->_wrapper->hybrid_set()->size(), 1);
+    ASSERT_NE(merger->debug_string().find("received_producer_num: 1"), 
std::string::npos);
 }
 
 TEST_F(RuntimeFilterMergerTest, basic) {
diff --git a/be/test/exec/runtime_filter/runtime_filter_producer_test.cpp 
b/be/test/exec/runtime_filter/runtime_filter_producer_test.cpp
index 42062eec3e4..87c697ba502 100644
--- a/be/test/exec/runtime_filter/runtime_filter_producer_test.cpp
+++ b/be/test/exec/runtime_filter/runtime_filter_producer_test.cpp
@@ -474,10 +474,9 @@ TEST_F(RuntimeFilterProducerTest, 
publish_mixed_targets_disabled) {
     ASSERT_NE(local_consumer->_wrapper, merge_consumer->_wrapper);
 }
 
-// A runtime filter whose only targets are local-merge (global RF mgr) 
consumers never hands its
-// wrapper to a plain local consumer of either instance, so once it reaches 
the merger nothing
-// else reads it: the merger may adopt it directly instead of cloning it.
-TEST_F(RuntimeFilterProducerTest, 
publish_local_merge_only_takes_ownership_without_local_consumer) {
+// Even without a plain local consumer, the merger keeps its own copy of each 
input.
+TEST_F(RuntimeFilterProducerTest,
+       publish_local_merge_only_uses_private_copy_without_local_consumer) {
     auto desc = TRuntimeFilterDescBuilder()
                         .set_build_bf_by_runtime_size(false)
                         .set_is_broadcast_join(false)
@@ -505,30 +504,40 @@ TEST_F(RuntimeFilterProducerTest, 
publish_local_merge_only_takes_ownership_witho
     
producer2->set_wrapper_state_and_ready_to_publish(RuntimeFilterWrapper::State::READY);
 
     auto wrapper = producer->wrapper();
+    auto wrapper2 = producer2->wrapper();
     
FAIL_IF_ERROR_OR_CATCH_EXCEPTION(producer->publish(_runtime_states[0].get(), 
true));
     std::shared_ptr<LocalMergeContext> context;
     FAIL_IF_ERROR_OR_CATCH_EXCEPTION(
             
_query_ctx->runtime_filter_mgr()->get_local_merge_context(desc.filter_id, 0, 
&context));
-    // The merger adopted the first producer's wrapper directly instead of 
cloning it: no plain
-    // local consumer of this instance exists to be affected.
-    ASSERT_EQ(context->merger->_wrapper, wrapper);
+    ASSERT_NE(context->merger->_wrapper, wrapper);
+    ASSERT_NE(context->merger->_wrapper->hybrid_set(), wrapper->hybrid_set());
     ASSERT_EQ(context->merger->_wrapper->hybrid_set()->size(), 1);
+    ASSERT_FALSE(context->merger->ready());
+    auto merged_wrapper = context->merger->_wrapper;
 
     
FAIL_IF_ERROR_OR_CATCH_EXCEPTION(producer2->publish(_runtime_states[1].get(), 
true));
     ASSERT_EQ(consumer->_rf_state, RuntimeFilterConsumer::State::READY);
-    ASSERT_EQ(consumer->_wrapper, context->merger->_wrapper);
+    ASSERT_EQ(consumer->_wrapper, merged_wrapper);
+    ASSERT_EQ(context->merger->_wrapper, merged_wrapper);
     ASSERT_EQ(consumer->_wrapper->hybrid_set()->size(), 2);
     int32_t five = 5;
     int32_t six = 6;
+    ASSERT_TRUE(merged_wrapper->hybrid_set()->find(&five));
+    ASSERT_TRUE(merged_wrapper->hybrid_set()->find(&six));
     ASSERT_TRUE(wrapper->hybrid_set()->find(&five));
-    ASSERT_TRUE(wrapper->hybrid_set()->find(&six));
+    ASSERT_FALSE(wrapper->hybrid_set()->find(&six));
+    ASSERT_EQ(wrapper->hybrid_set()->size(), 1);
+    ASSERT_NE(merged_wrapper->hybrid_set(), wrapper2->hybrid_set());
+    ASSERT_TRUE(wrapper2->hybrid_set()->find(&six));
+    ASSERT_FALSE(wrapper2->hybrid_set()->find(&five));
+    ASSERT_EQ(wrapper2->hybrid_set()->size(), 1);
+    ASSERT_EQ(wrapper->get_state(), RuntimeFilterWrapper::State::READY);
+    ASSERT_EQ(wrapper2->get_state(), RuntimeFilterWrapper::State::READY);
 }
 
-// A non-broadcast, non-runtime-sized filter with exactly one expected 
producer becomes ready on
-// that producer's only `merge_from` call and is never merged into again, so 
the merger may adopt
-// the producer's wrapper directly even though a plain local consumer of the 
same instance also
-// reads it.
-TEST_F(RuntimeFilterProducerTest, 
publish_single_producer_with_local_consumer_takes_ownership) {
+// The local consumer retains the producer wrapper; the merger owns a distinct 
copy even with one
+// expected producer.
+TEST_F(RuntimeFilterProducerTest, 
publish_single_producer_with_local_consumer_uses_private_copy) {
     auto desc = TRuntimeFilterDescBuilder()
                         .set_build_bf_by_runtime_size(false)
                         .set_is_broadcast_join(false)
@@ -558,18 +567,18 @@ TEST_F(RuntimeFilterProducerTest, 
publish_single_producer_with_local_consumer_ta
     ASSERT_EQ(local_consumer->_rf_state, RuntimeFilterConsumer::State::READY);
     ASSERT_EQ(local_consumer->_wrapper, wrapper);
     ASSERT_EQ(merge_consumer->_rf_state, RuntimeFilterConsumer::State::READY);
-    // The merger adopted the single producer's wrapper directly: the same 
object the plain local
-    // consumer also received, because the merger never merges another 
producer into it.
-    ASSERT_EQ(merge_consumer->_wrapper, wrapper);
+    ASSERT_NE(merge_consumer->_wrapper, wrapper);
+    ASSERT_NE(merge_consumer->_wrapper->hybrid_set(), wrapper->hybrid_set());
     int32_t five = 5;
     ASSERT_TRUE(wrapper->hybrid_set()->find(&five));
+    ASSERT_TRUE(merge_consumer->_wrapper->hybrid_set()->find(&five));
+    ASSERT_EQ(wrapper->hybrid_set()->size(), 1);
+    ASSERT_EQ(merge_consumer->_wrapper->hybrid_set()->size(), 1);
 }
 
-// A non-broadcast filter with only remote targets never hands its wrapper to 
a plain local
-// consumer: that only happens in the `!_has_remote_target` branch of 
`publish()`. Once such a
-// producer's wrapper reaches the local merger, no other reader remains, so 
the merger may take
-// it over directly instead of cloning it.
-TEST_F(RuntimeFilterProducerTest, publish_remote_target_merge_takes_ownership) 
{
+// A remote-target producer does not expose its wrapper to a local consumer, 
but the local merger
+// still makes a private copy before the producer releases its reference.
+TEST_F(RuntimeFilterProducerTest, 
publish_remote_target_merge_uses_private_copy) {
     auto desc = 
TRuntimeFilterDescBuilder().set_mode(false).set_is_broadcast_join(false).build();
 
     std::shared_ptr<RuntimeFilterProducer> producer;
@@ -593,11 +602,29 @@ TEST_F(RuntimeFilterProducerTest, 
publish_remote_target_merge_takes_ownership) {
     std::shared_ptr<LocalMergeContext> context;
     FAIL_IF_ERROR_OR_CATCH_EXCEPTION(
             
_query_ctx->runtime_filter_mgr()->get_local_merge_context(desc.filter_id, 0, 
&context));
-    // The merger adopted the producer's wrapper directly instead of cloning 
it.
-    ASSERT_EQ(context->merger->_wrapper, wrapper);
+    ASSERT_NE(context->merger->_wrapper, wrapper);
+    ASSERT_NE(context->merger->_wrapper->hybrid_set(), wrapper->hybrid_set());
     ASSERT_EQ(wrapper->hybrid_set()->size(), 1);
     int32_t five = 5;
+    int32_t six = 6;
     ASSERT_TRUE(wrapper->hybrid_set()->find(&five));
+    auto merged_wrapper = context->merger->_wrapper;
+    FAIL_IF_ERROR_OR_CATCH_EXCEPTION(producer2->init(1));
+    FAIL_IF_ERROR_OR_CATCH_EXCEPTION(
+            producer2->insert(ColumnHelper::create_column<DataTypeInt32>({6}), 
0));
+    
producer2->set_wrapper_state_and_ready_to_publish(RuntimeFilterWrapper::State::READY);
+    auto wrapper2 = producer2->wrapper();
+    bool ready = false;
+    
FAIL_IF_ERROR_OR_CATCH_EXCEPTION(context->merger->merge_from(producer2.get(), 
&ready));
+    ASSERT_TRUE(ready);
+    ASSERT_EQ(context->merger->_wrapper, merged_wrapper);
+    ASSERT_TRUE(merged_wrapper->hybrid_set()->find(&five));
+    ASSERT_TRUE(merged_wrapper->hybrid_set()->find(&six));
+    ASSERT_EQ(merged_wrapper->hybrid_set()->size(), 2);
+    ASSERT_EQ(wrapper->hybrid_set()->size(), 1);
+    ASSERT_FALSE(wrapper->hybrid_set()->find(&six));
+    ASSERT_EQ(wrapper2->hybrid_set()->size(), 1);
+    ASSERT_FALSE(wrapper2->hybrid_set()->find(&five));
 }
 
 // An IN_OR_BLOOM merger which is still an IN filter takes the bloom filter of 
a producer which
@@ -687,4 +714,4 @@ TEST_F(RuntimeFilterProducerTest, publish_release_wrapper) {
     ASSERT_EQ(producer->_wrapper, nullptr);
 }
 
-} // namespace doris
\ No newline at end of file
+} // namespace doris
diff --git a/be/test/exec/runtime_filter/runtime_filter_wrapper_test.cpp 
b/be/test/exec/runtime_filter/runtime_filter_wrapper_test.cpp
index fef8ed15ea9..67198957313 100644
--- a/be/test/exec/runtime_filter/runtime_filter_wrapper_test.cpp
+++ b/be/test/exec/runtime_filter/runtime_filter_wrapper_test.cpp
@@ -1399,16 +1399,14 @@ TEST_F(RuntimeFilterWrapperTest, 
TestMergeInWithBloomCopiesBloomFilter) {
             std::all_of(final_res.begin(), final_res.end(), [](uint8_t i) -> 
bool { return i; }));
 }
 
-// When the caller tells merge() that `other` is exclusively owned (e.g. an 
RPC-only filter with
-// no other reader, see `RuntimeFilterMergeControllerEntity::merge`), the 
bloom filter is taken
-// over directly instead of cloned.
-TEST_F(RuntimeFilterWrapperTest, 
TestMergeInWithBloomExclusivelyOwnedTakesOwnership) {
+// A null-aware IN input must be inserted into a private copy of the Bloom 
input.
+TEST_F(RuntimeFilterWrapperTest, TestMergeInWithBloomCopiesNullAwareInput) {
     using DataType = DataTypeInt32;
     auto make_params = [](int32_t max_in_num) {
         return RuntimeFilterParams {.filter_id = 0,
                                     .filter_type = 
RuntimeFilterType::IN_OR_BLOOM_FILTER,
                                     .column_return_type = 
PrimitiveType::TYPE_INT,
-                                    .null_aware = false,
+                                    .null_aware = true,
                                     .max_in_num = max_in_num,
                                     .runtime_bloom_filter_min_size = 64,
                                     .runtime_bloom_filter_max_size = 128,
@@ -1423,6 +1421,8 @@ TEST_F(RuntimeFilterWrapperTest, 
TestMergeInWithBloomExclusivelyOwnedTakesOwners
     std::vector<int> data_vector(10);
     std::iota(data_vector.begin(), data_vector.end(), 0);
     
ASSERT_TRUE(wrapper->insert(ColumnHelper::create_column<DataType>(data_vector), 
0).ok());
+    wrapper->hybrid_set()->insert((const void*)nullptr);
+    ASSERT_TRUE(wrapper->contain_null());
 
     auto bloom_params = make_params(8);
     auto bloom_wrapper = std::make_shared<RuntimeFilterWrapper>(&bloom_params);
@@ -1434,12 +1434,24 @@ TEST_F(RuntimeFilterWrapperTest, 
TestMergeInWithBloomExclusivelyOwnedTakesOwners
                         .ok());
     bloom_wrapper->set_state(RuntimeFilterWrapper::State::READY);
     auto bloom_filter_func_before_merge = bloom_wrapper->bloom_filter_func();
+    char* bloom_data = nullptr;
+    int bloom_len = 0;
+    bloom_filter_func_before_merge->get_data(&bloom_data, &bloom_len);
+    ASSERT_GT(bloom_len, 0);
+    const std::string snapshot(bloom_data, bloom_len);
+    ASSERT_FALSE(bloom_wrapper->contain_null());
 
-    ASSERT_TRUE(wrapper->merge(bloom_wrapper.get(), 
/*other_exclusively_owned=*/true).ok());
+    ASSERT_TRUE(wrapper->merge(bloom_wrapper.get()).ok());
     EXPECT_EQ(wrapper->get_real_type(), RuntimeFilterType::BLOOM_FILTER);
-    // No clone: the merger took over `other`'s bloom filter directly.
-    EXPECT_EQ(wrapper->bloom_filter_func(), bloom_filter_func_before_merge);
-    EXPECT_EQ(wrapper->bloom_filter_func(), 
bloom_wrapper->bloom_filter_func());
+    EXPECT_NE(wrapper->bloom_filter_func(), bloom_filter_func_before_merge);
+    char* merged_data = nullptr;
+    int merged_len = 0;
+    wrapper->bloom_filter_func()->get_data(&merged_data, &merged_len);
+    EXPECT_NE(merged_data, bloom_data);
+    EXPECT_EQ(merged_len, bloom_len);
+    EXPECT_EQ(snapshot, std::string(bloom_data, bloom_len));
+    EXPECT_FALSE(bloom_wrapper->contain_null());
+    EXPECT_TRUE(wrapper->contain_null());
 
     std::vector<int> final_data_vector(20);
     std::iota(final_data_vector.begin(), final_data_vector.end(), 0);
@@ -1465,6 +1477,23 @@ TEST_F(RuntimeFilterWrapperTest, TestClone) {
                                     .build_bf_by_runtime_size = false,
                                     .bloom_filter_size_calculated_by_ndv = 
true};
     };
+    {
+        // An initialized empty IN filter still needs a separately mutable set.
+        auto params = make_params(RuntimeFilterType::IN_FILTER, false, 8);
+        auto wrapper = std::make_shared<RuntimeFilterWrapper>(&params);
+        ASSERT_TRUE(wrapper->init(0).ok());
+        wrapper->set_state(RuntimeFilterWrapper::State::READY);
+        std::shared_ptr<RuntimeFilterWrapper> cloned;
+        ASSERT_TRUE(wrapper->clone(&cloned).ok());
+        EXPECT_NE(cloned, wrapper);
+        EXPECT_NE(cloned->hybrid_set(), wrapper->hybrid_set());
+        EXPECT_EQ(cloned->hybrid_set()->size(), 0);
+        EXPECT_FALSE(cloned->contain_null());
+        int32_t one = 1;
+        cloned->hybrid_set()->insert(&one);
+        EXPECT_EQ(wrapper->hybrid_set()->size(), 0);
+        EXPECT_EQ(cloned->hybrid_set()->size(), 1);
+    }
     {
         // In filter, the copy starts with an empty bucket prune cache.
         auto params = make_params(RuntimeFilterType::IN_FILTER, true, 8);


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to