This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/branch-4.2 by this push:
     new 91ea288e0ed branch-4.2: [fix](cloud) Fix rowset notification cleanup 
and update cloud load defaults #70218 (#70455)
91ea288e0ed is described below

commit 91ea288e0edcc1eb3bdb50f347a7deea3cba20f9
Author: bobhan1 <[email protected]>
AuthorDate: Sat Nov 21 12:10:33 2026 +0800

    branch-4.2: [fix](cloud) Fix rowset notification cleanup and update cloud 
load defaults #70218 (#70455)
    
    ### What problem does this PR solve?
    
    Issue Number: None
    
    Related PR: #70218
    
    Backport #70218 (`08231e3a6359082264c88e6fe2c8050e911e5236`) to
    `branch-4.2`.
    
    Completed committed-rowset notifications retain task signatures, and
    consumed empty-rowset markers remain cached. Release signatures on every
    callback exit and give empty and non-empty rowsets the same removal and
    expiration lifecycle. Enable the existing FE/BE notification settings by
    default, disable forced initial MoW rowset synchronization by default,
    and remove the six redundant Cloud p0/p1 configuration overrides.
    
    ### Branch-specific adaptations
    
    - Keep the existing 4.2 configuration layout. Random segment IDs and
    Table Stream settings from master are outside this backport.
    - Omit the fixture-only update to `CloudGroupRowsetBuilderWriterTest`:
    that test and its corresponding feature are absent from `branch-4.2`.
    The notification and cache regression tests are identical to the
    upstream versions.
    
    ### Testing
    
    - `./run-fe-ut.sh --run
    org.apache.doris.cloud.transaction.CloudGlobalTransactionMgrTest`: 33
    tests passed, 0 failures/errors/skips; Maven `BUILD SUCCESS`.
    - `./run-be-ut.sh --run
    
--filter='CloudCommittedRSMgrTest.*:EnabledAndDisabled/CloudCommittedRowsetTaskTest.*:TaskWorkerPoolTest.*:CloudTabletApplyVisiblePendingTest.*:*DeltaWriterCancelTest.*'
    -j100`: 44 ASAN tests passed, including both enabled/disabled
    signature-release cases, empty-rowset removal, expiration refresh,
    pending-rowset visibility, and local/cloud cancellation.
    - Clang-format v16 passed for all six changed C++ files.
    - Verified that all six removed pipeline overrides equal the new
    defaults and that all other parsed settings remain unchanged.
    - `git diff --check upstream/branch-4.2...HEAD` passed.
    - Cloud end-to-end regression and a standalone production BE build were
    not run.
    
    ### Release note
    
    Cloud committed-rowset notifications now default to enabled: FE
    `enable_notify_be_after_load_txn_commit=true` and BE
    `enable_cloud_make_rs_visible_on_be=true`. Completed notifications
    release their task signatures, and consumed empty-rowset entries are
    removed promptly. BE `cloud_mow_sync_rowsets_when_load_txn_begin` now
    defaults to `false`.
    
    The original PR's behavior and upgrade considerations apply: skipping
    initial synchronization can enlarge the existing local-snapshot
    freshness window. During rolling upgrades, keep FE notifications
    disabled until all receiving BEs contain the signature-cleanup fix.
    
    ### Check List (For Author)
    
    - Test: targeted BE/FE unit tests and configuration checks described
    above.
    - Behavior changed: Yes; the three configuration defaults change as
    described above.
    - Does this need documentation: Yes; the default changes should be
    reflected in configuration documentation and release notes.
    
    ### AI Disclosure
    
    - Tool: Codex (GPT-6).
    - Backport conflict resolution and this description were prepared with
    AI assistance; the implementation originates from #70218.
    - Human review: Pending.
---
 be/src/agent/task_worker_pool.cpp                  |  1 +
 be/src/cloud/cloud_committed_rs_mgr.cpp            | 45 +++++++------------
 be/src/cloud/cloud_committed_rs_mgr.h              |  2 +-
 be/src/common/config.cpp                           |  4 +-
 be/test/agent/task_worker_pool_test.cpp            | 38 ++++++++++++++++
 be/test/cloud/cloud_committed_rs_mgr_test.cpp      | 50 ++++++++++++++++++++++
 .../main/java/org/apache/doris/common/Config.java  |  2 +-
 .../pipeline/cloud_p0/conf/be_custom.conf          |  2 -
 .../pipeline/cloud_p0/conf/fe_custom.conf          |  3 +-
 .../pipeline/cloud_p1/conf/be_custom.conf          |  2 -
 .../pipeline/cloud_p1/conf/fe_custom.conf          |  2 -
 11 files changed, 109 insertions(+), 42 deletions(-)

diff --git a/be/src/agent/task_worker_pool.cpp 
b/be/src/agent/task_worker_pool.cpp
index c9e7ab75c7e..dd1e4d3a81a 100644
--- a/be/src/agent/task_worker_pool.cpp
+++ b/be/src/agent/task_worker_pool.cpp
@@ -2571,6 +2571,7 @@ void calc_delete_bitmap_callback(CloudStorageEngine& 
engine, const TAgentTaskReq
 
 void make_cloud_committed_rs_visible_callback(CloudStorageEngine& engine,
                                               const TAgentTaskRequest& req) {
+    Defer remove_task = [&] { remove_task_info(req.task_type, req.signature); 
};
     if (!config::enable_cloud_make_rs_visible_on_be) {
         return;
     }
diff --git a/be/src/cloud/cloud_committed_rs_mgr.cpp 
b/be/src/cloud/cloud_committed_rs_mgr.cpp
index 3d96b7ca7f7..989b1864b10 100644
--- a/be/src/cloud/cloud_committed_rs_mgr.cpp
+++ b/be/src/cloud/cloud_committed_rs_mgr.cpp
@@ -53,22 +53,21 @@ void CloudCommittedRSMgr::add_committed_rowset(int64_t 
txn_id, int64_t tablet_id
                     .count() +
             config::tablet_txn_info_min_expired_seconds;
     expiration_time = std::max(txn_expiration_min, expiration_time);
-    std::unique_lock<std::shared_mutex> wlock(_rwlock);
-    TxnTabletKey key(txn_id, tablet_id);
-    _committed_rs_map.insert_or_assign(key, CommittedRowsetValue(rowset_meta, 
expiration_time));
-    _expiration_map.emplace(expiration_time, key);
-    LOG(INFO) << "add pending rowset, txn_id=" << txn_id << ", tablet_id=" << 
tablet_id
-              << ", rowset_id=" << rowset_meta->rowset_id().to_string()
-              << ", expiration_time=" << expiration_time;
+    {
+        std::unique_lock<std::shared_mutex> wlock(_rwlock);
+        TxnTabletKey key(txn_id, tablet_id);
+        _committed_rs_map.insert_or_assign(key, 
CommittedRowsetValue(rowset_meta, expiration_time));
+        _expiration_map.emplace(expiration_time, key);
+    }
+    VLOG_DEBUG << "add pending rowset, txn_id=" << txn_id << ", tablet_id=" << 
tablet_id
+               << ", rowset_id=" << (rowset_meta ? 
rowset_meta->rowset_id().to_string() : "empty")
+               << ", expiration_time=" << expiration_time;
 }
 
 Result<std::pair<RowsetMetaSharedPtr, int64_t>> 
CloudCommittedRSMgr::get_committed_rowset(
         int64_t txn_id, int64_t tablet_id) {
     std::shared_lock<std::shared_mutex> rlock(_rwlock);
     TxnTabletKey key(txn_id, tablet_id);
-    if (auto it = _empty_rowset_markers.find(key); it != 
_empty_rowset_markers.end()) {
-        return std::make_pair(nullptr, it->second);
-    }
     auto iter = _committed_rs_map.find(key);
     if (iter == _committed_rs_map.end()) {
         return ResultError(Status::Error<ErrorCode::NOT_FOUND>(
@@ -90,8 +89,9 @@ void CloudCommittedRSMgr::remove_expired_committed_rowsets() {
 
     while (!_expiration_map.empty()) {
         auto iter = _expiration_map.begin();
-        if (!_committed_rs_map.contains(iter->second) &&
-            !_empty_rowset_markers.contains(iter->second)) {
+        // Consumed entries leave expiration records behind until this cleanup 
pass.
+        auto it_rs = _committed_rs_map.find(iter->second);
+        if (it_rs == _committed_rs_map.end()) {
             _expiration_map.erase(iter);
             continue;
         }
@@ -103,33 +103,18 @@ void 
CloudCommittedRSMgr::remove_expired_committed_rowsets() {
         auto key = iter->second;
         _expiration_map.erase(iter);
 
-        auto it_rs = _committed_rs_map.find(key);
-        if (it_rs != _committed_rs_map.end() && it_rs->second.expiration_time 
== expiration_time) {
+        // A refreshed entry can outlive this older expiration record.
+        if (it_rs->second.expiration_time == expiration_time) {
             _committed_rs_map.erase(it_rs);
             LOG(INFO) << "clean expired pending cloud rowset, txn_id=" << 
key.txn_id
                       << ", tablet_id=" << key.tablet_id << ", 
expiration_time=" << expiration_time;
         }
-        auto it_empty = _empty_rowset_markers.find(key);
-        if (it_empty != _empty_rowset_markers.end() && it_empty->second == 
expiration_time) {
-            _empty_rowset_markers.erase(it_empty);
-            LOG(INFO) << "clean expired empty rowset marker, txn_id=" << 
key.txn_id
-                      << ", tablet_id=" << key.tablet_id << ", 
expiration_time=" << expiration_time;
-        }
     }
 }
 
 void CloudCommittedRSMgr::mark_empty_rowset(int64_t txn_id, int64_t tablet_id,
                                             int64_t txn_expiration) {
-    int64_t txn_expiration_min =
-            
duration_cast<std::chrono::seconds>(std::chrono::system_clock::now().time_since_epoch())
-                    .count() +
-            config::tablet_txn_info_min_expired_seconds;
-    txn_expiration = std::max(txn_expiration_min, txn_expiration);
-
-    std::unique_lock<std::shared_mutex> wlock(_rwlock);
-    TxnTabletKey txn_key(txn_id, tablet_id);
-    _empty_rowset_markers.insert_or_assign(txn_key, txn_expiration);
-    _expiration_map.emplace(txn_expiration, txn_key);
+    add_committed_rowset(txn_id, tablet_id, nullptr, txn_expiration);
 }
 
 void CloudCommittedRSMgr::_clean_thread_callback() {
diff --git a/be/src/cloud/cloud_committed_rs_mgr.h 
b/be/src/cloud/cloud_committed_rs_mgr.h
index 33af0ba3979..b83734e8b25 100644
--- a/be/src/cloud/cloud_committed_rs_mgr.h
+++ b/be/src/cloud/cloud_committed_rs_mgr.h
@@ -67,6 +67,7 @@ private:
     };
 
     struct CommittedRowsetValue {
+        // A null meta marks an empty rowset whose metadata was not written to 
MS.
         RowsetMetaSharedPtr rowset_meta;
         int64_t expiration_time; // seconds since epoch
 
@@ -78,7 +79,6 @@ private:
     std::map<TxnTabletKey, CommittedRowsetValue> _committed_rs_map;
     // Multimap for efficient expiration cleanup: expiration_time -> <txn_id, 
tablet_id>
     std::multimap<int64_t, TxnTabletKey> _expiration_map;
-    std::map<TxnTabletKey, int64_t /* expiration_time */> 
_empty_rowset_markers;
     std::shared_mutex _rwlock;
     std::shared_ptr<Thread> _clean_thread;
     CountDownLatch _stop_latch;
diff --git a/be/src/common/config.cpp b/be/src/common/config.cpp
index 917cbfafa3a..1fbf06a8214 100644
--- a/be/src/common/config.cpp
+++ b/be/src/common/config.cpp
@@ -2050,9 +2050,9 @@ DEFINE_mInt32(concurrency_stats_dump_interval_ms, "100");
 DEFINE_Validator(concurrency_stats_dump_interval_ms,
                  [](const int32_t config) -> bool { return config >= 10; });
 
-DEFINE_mBool(cloud_mow_sync_rowsets_when_load_txn_begin, "true");
+DEFINE_mBool(cloud_mow_sync_rowsets_when_load_txn_begin, "false");
 
-DEFINE_mBool(enable_cloud_make_rs_visible_on_be, "false");
+DEFINE_mBool(enable_cloud_make_rs_visible_on_be, "true");
 DEFINE_mInt32(file_handles_deplenish_frequency_times, "3");
 
 // clang-format off
diff --git a/be/test/agent/task_worker_pool_test.cpp 
b/be/test/agent/task_worker_pool_test.cpp
index b8b2de0fc4b..8323ad1bd66 100644
--- a/be/test/agent/task_worker_pool_test.cpp
+++ b/be/test/agent/task_worker_pool_test.cpp
@@ -27,10 +27,13 @@
 
 #include "agent/agent_server.h"
 #include "cloud/cloud_storage_engine.h"
+#include "common/config.h"
 #include "runtime/cluster_info.h"
 #include "runtime/exec_env.h"
 #include "storage/options.h"
 #include "storage/storage_engine.h"
+#include "util/defer_op.h"
+#include "util/threadpool.h"
 
 namespace doris {
 
@@ -106,6 +109,41 @@ TEST(TaskWorkerPoolTest, PreSubmitCallbackWithDedup) {
     EXPECT_EQ(callback_count.load(), 1);
 }
 
+class CloudCommittedRowsetTaskTest : public testing::TestWithParam<bool> {};
+
+TEST_P(CloudCommittedRowsetTaskTest, ReleasesSignatureAfterCallback) {
+    const bool original_enabled = config::enable_cloud_make_rs_visible_on_be;
+    Defer restore_config = [&] { config::enable_cloud_make_rs_visible_on_be = 
original_enabled; };
+    config::enable_cloud_make_rs_visible_on_be = GetParam();
+
+    CloudStorageEngine engine(EngineOptions {});
+    std::atomic_int callback_count {0};
+    TaskWorkerPool workers("make_cloud_committed_rs_visible_test", 1, 
[&](const auto& task) {
+        make_cloud_committed_rs_visible_callback(engine, task);
+        ++callback_count;
+    });
+
+    TAgentTaskRequest task;
+    task.__set_task_type(TTaskType::MAKE_CLOUD_COMMITTED_RS_VISIBLE);
+    task.__set_signature(GetParam() ? 5975401 : 5975400);
+    task.make_cloud_tmp_rs_visible_req.__set_txn_id(task.signature);
+    // A tablet absent from the local cache is skipped without fetching 
metadata from MS.
+    task.make_cloud_tmp_rs_visible_req.__set_tablet_ids({59754});
+
+    ASSERT_TRUE(workers.submit_task(task).ok());
+    workers._thread_pool->wait();
+    ASSERT_EQ(callback_count.load(), 1);
+
+    // A completed notification must not suppress a later task with the same 
signature,
+    // including when the feature was disabled before the first callback ran.
+    ASSERT_TRUE(workers.submit_task(task).ok());
+    workers._thread_pool->wait();
+    EXPECT_EQ(callback_count.load(), 2);
+}
+
+INSTANTIATE_TEST_SUITE_P(EnabledAndDisabled, CloudCommittedRowsetTaskTest,
+                         testing::Values(true, false));
+
 TEST(TaskWorkerPoolTest, PriorTaskWorkerPool) {
     std::atomic_int normal_count {0};
     std::atomic_int high_prior_count {0};
diff --git a/be/test/cloud/cloud_committed_rs_mgr_test.cpp 
b/be/test/cloud/cloud_committed_rs_mgr_test.cpp
index 9df6597b8a6..791f168919e 100644
--- a/be/test/cloud/cloud_committed_rs_mgr_test.cpp
+++ b/be/test/cloud/cloud_committed_rs_mgr_test.cpp
@@ -25,6 +25,7 @@
 
 #include "cloud/config.h"
 #include "storage/rowset/rowset_meta.h"
+#include "util/defer_op.h"
 
 namespace doris {
 
@@ -179,6 +180,55 @@ TEST_F(CloudCommittedRSMgrTest, 
TestMarkAndCheckEmptyRowset) {
     EXPECT_EQ(retrieved_expiration, txn_expiration);
 }
 
+TEST_F(CloudCommittedRSMgrTest, TestRemoveEmptyRowset) {
+    const int64_t expiration_time = current_time_seconds() + 3600;
+    _mgr->mark_empty_rowset(1000, 2000, expiration_time);
+    _mgr->mark_empty_rowset(1000, 2001, expiration_time);
+    _mgr->mark_empty_rowset(1001, 2000, expiration_time);
+
+    _mgr->remove_committed_rowset(1000, 2000);
+    auto result = _mgr->get_committed_rowset(1000, 2000);
+    ASSERT_FALSE(result.has_value());
+    EXPECT_TRUE(result.error().is<ErrorCode::NOT_FOUND>());
+    EXPECT_TRUE(_mgr->get_committed_rowset(1000, 2001).has_value());
+    EXPECT_TRUE(_mgr->get_committed_rowset(1001, 2000).has_value());
+
+    // Repeated removal is harmless, and a consumed empty marker cannot mask a 
new entry.
+    _mgr->remove_committed_rowset(1000, 2000);
+    auto rowset_meta = create_rowset_meta(2000, 1000);
+    _mgr->add_committed_rowset(1000, 2000, rowset_meta, expiration_time);
+    _mgr->remove_expired_committed_rowsets();
+    result = _mgr->get_committed_rowset(1000, 2000);
+    ASSERT_TRUE(result.has_value());
+    EXPECT_EQ(result->first, rowset_meta);
+}
+
+TEST_F(CloudCommittedRSMgrTest, TestRefreshExpiration) {
+    const auto original_min_expired_seconds = 
config::tablet_txn_info_min_expired_seconds;
+    Defer restore_config = [&] {
+        config::tablet_txn_info_min_expired_seconds = 
original_min_expired_seconds;
+    };
+    config::tablet_txn_info_min_expired_seconds = -100;
+    const int64_t now = current_time_seconds();
+
+    // An old expiration index entry must not remove either kind of refreshed 
cache entry.
+    auto rowset_meta = create_rowset_meta(2000, 1000);
+    _mgr->add_committed_rowset(1000, 2000, rowset_meta, now - 10);
+    _mgr->mark_empty_rowset(1000, 2001, now - 10);
+    _mgr->add_committed_rowset(1000, 2000, rowset_meta, now + 3600);
+    _mgr->mark_empty_rowset(1000, 2001, now + 3600);
+    _mgr->remove_expired_committed_rowsets();
+
+    auto result = _mgr->get_committed_rowset(1000, 2000);
+    ASSERT_TRUE(result.has_value());
+    EXPECT_EQ(result->first, rowset_meta);
+    EXPECT_EQ(result->second, now + 3600);
+    result = _mgr->get_committed_rowset(1000, 2001);
+    ASSERT_TRUE(result.has_value());
+    EXPECT_EQ(result->first, nullptr);
+    EXPECT_EQ(result->second, now + 3600);
+}
+
 TEST_F(CloudCommittedRSMgrTest, TestEmptyRowsetExpiration) {
     // Save original config value
     int32_t original_min_expired_seconds = 
config::tablet_txn_info_min_expired_seconds;
diff --git a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java 
b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
index 6c201e1ad4e..69a8840b7e9 100644
--- a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
+++ b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
@@ -4058,7 +4058,7 @@ public class Config extends ConfigBase {
     public static long mow_get_ms_lock_retry_backoff_interval = 80;
 
     @ConfField(mutable = true, masterOnly = true)
-    public static boolean enable_notify_be_after_load_txn_commit = false;
+    public static boolean enable_notify_be_after_load_txn_commit = true;
 
     // ATTN: DONOT add any config not related to cloud mode here
     // ATTN: DONOT add any config not related to cloud mode here
diff --git a/regression-test/pipeline/cloud_p0/conf/be_custom.conf 
b/regression-test/pipeline/cloud_p0/conf/be_custom.conf
index b2aef18c55c..3d5ef2e3ad1 100644
--- a/regression-test/pipeline/cloud_p0/conf/be_custom.conf
+++ b/regression-test/pipeline/cloud_p0/conf/be_custom.conf
@@ -59,8 +59,6 @@ enable_segment_rows_consistency_check=true
 enable_segment_rows_check_core=true
 fail_when_segment_rows_not_in_rowset_meta=true
 
-enable_cloud_make_rs_visible_on_be=true
-cloud_mow_sync_rowsets_when_load_txn_begin=false
 # enable to use python udf
 enable_python_udf_support=true
 # The CI image supplies a Python venv; conda is not installed in this runtime.
diff --git a/regression-test/pipeline/cloud_p0/conf/fe_custom.conf 
b/regression-test/pipeline/cloud_p0/conf/fe_custom.conf
index d6c2070aa93..2e4663e7ee6 100644
--- a/regression-test/pipeline/cloud_p0/conf/fe_custom.conf
+++ b/regression-test/pipeline/cloud_p0/conf/fe_custom.conf
@@ -48,5 +48,4 @@ enable_advance_next_id = true
 check_table_lock_leaky = true
 enable_outfile_to_local=true
 
-enable_notify_be_after_load_txn_commit=true
-max_bucket_num_per_partition=512
\ No newline at end of file
+max_bucket_num_per_partition=512
diff --git a/regression-test/pipeline/cloud_p1/conf/be_custom.conf 
b/regression-test/pipeline/cloud_p1/conf/be_custom.conf
index 0a9503ae878..8606b8aa199 100644
--- a/regression-test/pipeline/cloud_p1/conf/be_custom.conf
+++ b/regression-test/pipeline/cloud_p1/conf/be_custom.conf
@@ -41,8 +41,6 @@ enable_prefill_all_dbm_agg_cache_after_compaction=true
 enable_segment_rows_consistency_check=true
 enable_segment_rows_check_core=true
 
-enable_cloud_make_rs_visible_on_be=true
-cloud_mow_sync_rowsets_when_load_txn_begin=false
 # enable to use python udf
 enable_python_udf_support=true
 python_env_mode=conda
diff --git a/regression-test/pipeline/cloud_p1/conf/fe_custom.conf 
b/regression-test/pipeline/cloud_p1/conf/fe_custom.conf
index 157f3a07a09..b91a4ed6d38 100644
--- a/regression-test/pipeline/cloud_p1/conf/fe_custom.conf
+++ b/regression-test/pipeline/cloud_p1/conf/fe_custom.conf
@@ -37,5 +37,3 @@ enable_advance_next_id = true
 
 arrow_flight_sql_port = 8081
 enable_job_schedule_second_for_test = true
-
-enable_notify_be_after_load_txn_commit=true


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to