This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-4.2 by this push:
new 91ea288e0ed branch-4.2: [fix](cloud) Fix rowset notification cleanup
and update cloud load defaults #70218 (#70455)
91ea288e0ed is described below
commit 91ea288e0edcc1eb3bdb50f347a7deea3cba20f9
Author: bobhan1 <[email protected]>
AuthorDate: Sat Nov 21 12:10:33 2026 +0800
branch-4.2: [fix](cloud) Fix rowset notification cleanup and update cloud
load defaults #70218 (#70455)
### What problem does this PR solve?
Issue Number: None
Related PR: #70218
Backport #70218 (`08231e3a6359082264c88e6fe2c8050e911e5236`) to
`branch-4.2`.
Completed committed-rowset notifications retain task signatures, and
consumed empty-rowset markers remain cached. Release signatures on every
callback exit and give empty and non-empty rowsets the same removal and
expiration lifecycle. Enable the existing FE/BE notification settings by
default, disable forced initial MoW rowset synchronization by default,
and remove the six redundant Cloud p0/p1 configuration overrides.
### Branch-specific adaptations
- Keep the existing 4.2 configuration layout. Random segment IDs and
Table Stream settings from master are outside this backport.
- Omit the fixture-only update to `CloudGroupRowsetBuilderWriterTest`:
that test and its corresponding feature are absent from `branch-4.2`.
The notification and cache regression tests are identical to the
upstream versions.
### Testing
- `./run-fe-ut.sh --run
org.apache.doris.cloud.transaction.CloudGlobalTransactionMgrTest`: 33
tests passed, 0 failures/errors/skips; Maven `BUILD SUCCESS`.
- `./run-be-ut.sh --run
--filter='CloudCommittedRSMgrTest.*:EnabledAndDisabled/CloudCommittedRowsetTaskTest.*:TaskWorkerPoolTest.*:CloudTabletApplyVisiblePendingTest.*:*DeltaWriterCancelTest.*'
-j100`: 44 ASAN tests passed, including both enabled/disabled
signature-release cases, empty-rowset removal, expiration refresh,
pending-rowset visibility, and local/cloud cancellation.
- Clang-format v16 passed for all six changed C++ files.
- Verified that all six removed pipeline overrides equal the new
defaults and that all other parsed settings remain unchanged.
- `git diff --check upstream/branch-4.2...HEAD` passed.
- Cloud end-to-end regression and a standalone production BE build were
not run.
### Release note
Cloud committed-rowset notifications now default to enabled: FE
`enable_notify_be_after_load_txn_commit=true` and BE
`enable_cloud_make_rs_visible_on_be=true`. Completed notifications
release their task signatures, and consumed empty-rowset entries are
removed promptly. BE `cloud_mow_sync_rowsets_when_load_txn_begin` now
defaults to `false`.
The original PR's behavior and upgrade considerations apply: skipping
initial synchronization can enlarge the existing local-snapshot
freshness window. During rolling upgrades, keep FE notifications
disabled until all receiving BEs contain the signature-cleanup fix.
### Check List (For Author)
- Test: targeted BE/FE unit tests and configuration checks described
above.
- Behavior changed: Yes; the three configuration defaults change as
described above.
- Does this need documentation: Yes; the default changes should be
reflected in configuration documentation and release notes.
### AI Disclosure
- Tool: Codex (GPT-6).
- Backport conflict resolution and this description were prepared with
AI assistance; the implementation originates from #70218.
- Human review: Pending.
---
be/src/agent/task_worker_pool.cpp | 1 +
be/src/cloud/cloud_committed_rs_mgr.cpp | 45 +++++++------------
be/src/cloud/cloud_committed_rs_mgr.h | 2 +-
be/src/common/config.cpp | 4 +-
be/test/agent/task_worker_pool_test.cpp | 38 ++++++++++++++++
be/test/cloud/cloud_committed_rs_mgr_test.cpp | 50 ++++++++++++++++++++++
.../main/java/org/apache/doris/common/Config.java | 2 +-
.../pipeline/cloud_p0/conf/be_custom.conf | 2 -
.../pipeline/cloud_p0/conf/fe_custom.conf | 3 +-
.../pipeline/cloud_p1/conf/be_custom.conf | 2 -
.../pipeline/cloud_p1/conf/fe_custom.conf | 2 -
11 files changed, 109 insertions(+), 42 deletions(-)
diff --git a/be/src/agent/task_worker_pool.cpp
b/be/src/agent/task_worker_pool.cpp
index c9e7ab75c7e..dd1e4d3a81a 100644
--- a/be/src/agent/task_worker_pool.cpp
+++ b/be/src/agent/task_worker_pool.cpp
@@ -2571,6 +2571,7 @@ void calc_delete_bitmap_callback(CloudStorageEngine&
engine, const TAgentTaskReq
void make_cloud_committed_rs_visible_callback(CloudStorageEngine& engine,
const TAgentTaskRequest& req) {
+ Defer remove_task = [&] { remove_task_info(req.task_type, req.signature);
};
if (!config::enable_cloud_make_rs_visible_on_be) {
return;
}
diff --git a/be/src/cloud/cloud_committed_rs_mgr.cpp
b/be/src/cloud/cloud_committed_rs_mgr.cpp
index 3d96b7ca7f7..989b1864b10 100644
--- a/be/src/cloud/cloud_committed_rs_mgr.cpp
+++ b/be/src/cloud/cloud_committed_rs_mgr.cpp
@@ -53,22 +53,21 @@ void CloudCommittedRSMgr::add_committed_rowset(int64_t
txn_id, int64_t tablet_id
.count() +
config::tablet_txn_info_min_expired_seconds;
expiration_time = std::max(txn_expiration_min, expiration_time);
- std::unique_lock<std::shared_mutex> wlock(_rwlock);
- TxnTabletKey key(txn_id, tablet_id);
- _committed_rs_map.insert_or_assign(key, CommittedRowsetValue(rowset_meta,
expiration_time));
- _expiration_map.emplace(expiration_time, key);
- LOG(INFO) << "add pending rowset, txn_id=" << txn_id << ", tablet_id=" <<
tablet_id
- << ", rowset_id=" << rowset_meta->rowset_id().to_string()
- << ", expiration_time=" << expiration_time;
+ {
+ std::unique_lock<std::shared_mutex> wlock(_rwlock);
+ TxnTabletKey key(txn_id, tablet_id);
+ _committed_rs_map.insert_or_assign(key,
CommittedRowsetValue(rowset_meta, expiration_time));
+ _expiration_map.emplace(expiration_time, key);
+ }
+ VLOG_DEBUG << "add pending rowset, txn_id=" << txn_id << ", tablet_id=" <<
tablet_id
+ << ", rowset_id=" << (rowset_meta ?
rowset_meta->rowset_id().to_string() : "empty")
+ << ", expiration_time=" << expiration_time;
}
Result<std::pair<RowsetMetaSharedPtr, int64_t>>
CloudCommittedRSMgr::get_committed_rowset(
int64_t txn_id, int64_t tablet_id) {
std::shared_lock<std::shared_mutex> rlock(_rwlock);
TxnTabletKey key(txn_id, tablet_id);
- if (auto it = _empty_rowset_markers.find(key); it !=
_empty_rowset_markers.end()) {
- return std::make_pair(nullptr, it->second);
- }
auto iter = _committed_rs_map.find(key);
if (iter == _committed_rs_map.end()) {
return ResultError(Status::Error<ErrorCode::NOT_FOUND>(
@@ -90,8 +89,9 @@ void CloudCommittedRSMgr::remove_expired_committed_rowsets() {
while (!_expiration_map.empty()) {
auto iter = _expiration_map.begin();
- if (!_committed_rs_map.contains(iter->second) &&
- !_empty_rowset_markers.contains(iter->second)) {
+ // Consumed entries leave expiration records behind until this cleanup
pass.
+ auto it_rs = _committed_rs_map.find(iter->second);
+ if (it_rs == _committed_rs_map.end()) {
_expiration_map.erase(iter);
continue;
}
@@ -103,33 +103,18 @@ void
CloudCommittedRSMgr::remove_expired_committed_rowsets() {
auto key = iter->second;
_expiration_map.erase(iter);
- auto it_rs = _committed_rs_map.find(key);
- if (it_rs != _committed_rs_map.end() && it_rs->second.expiration_time
== expiration_time) {
+ // A refreshed entry can outlive this older expiration record.
+ if (it_rs->second.expiration_time == expiration_time) {
_committed_rs_map.erase(it_rs);
LOG(INFO) << "clean expired pending cloud rowset, txn_id=" <<
key.txn_id
<< ", tablet_id=" << key.tablet_id << ",
expiration_time=" << expiration_time;
}
- auto it_empty = _empty_rowset_markers.find(key);
- if (it_empty != _empty_rowset_markers.end() && it_empty->second ==
expiration_time) {
- _empty_rowset_markers.erase(it_empty);
- LOG(INFO) << "clean expired empty rowset marker, txn_id=" <<
key.txn_id
- << ", tablet_id=" << key.tablet_id << ",
expiration_time=" << expiration_time;
- }
}
}
void CloudCommittedRSMgr::mark_empty_rowset(int64_t txn_id, int64_t tablet_id,
int64_t txn_expiration) {
- int64_t txn_expiration_min =
-
duration_cast<std::chrono::seconds>(std::chrono::system_clock::now().time_since_epoch())
- .count() +
- config::tablet_txn_info_min_expired_seconds;
- txn_expiration = std::max(txn_expiration_min, txn_expiration);
-
- std::unique_lock<std::shared_mutex> wlock(_rwlock);
- TxnTabletKey txn_key(txn_id, tablet_id);
- _empty_rowset_markers.insert_or_assign(txn_key, txn_expiration);
- _expiration_map.emplace(txn_expiration, txn_key);
+ add_committed_rowset(txn_id, tablet_id, nullptr, txn_expiration);
}
void CloudCommittedRSMgr::_clean_thread_callback() {
diff --git a/be/src/cloud/cloud_committed_rs_mgr.h
b/be/src/cloud/cloud_committed_rs_mgr.h
index 33af0ba3979..b83734e8b25 100644
--- a/be/src/cloud/cloud_committed_rs_mgr.h
+++ b/be/src/cloud/cloud_committed_rs_mgr.h
@@ -67,6 +67,7 @@ private:
};
struct CommittedRowsetValue {
+ // A null meta marks an empty rowset whose metadata was not written to
MS.
RowsetMetaSharedPtr rowset_meta;
int64_t expiration_time; // seconds since epoch
@@ -78,7 +79,6 @@ private:
std::map<TxnTabletKey, CommittedRowsetValue> _committed_rs_map;
// Multimap for efficient expiration cleanup: expiration_time -> <txn_id,
tablet_id>
std::multimap<int64_t, TxnTabletKey> _expiration_map;
- std::map<TxnTabletKey, int64_t /* expiration_time */>
_empty_rowset_markers;
std::shared_mutex _rwlock;
std::shared_ptr<Thread> _clean_thread;
CountDownLatch _stop_latch;
diff --git a/be/src/common/config.cpp b/be/src/common/config.cpp
index 917cbfafa3a..1fbf06a8214 100644
--- a/be/src/common/config.cpp
+++ b/be/src/common/config.cpp
@@ -2050,9 +2050,9 @@ DEFINE_mInt32(concurrency_stats_dump_interval_ms, "100");
DEFINE_Validator(concurrency_stats_dump_interval_ms,
[](const int32_t config) -> bool { return config >= 10; });
-DEFINE_mBool(cloud_mow_sync_rowsets_when_load_txn_begin, "true");
+DEFINE_mBool(cloud_mow_sync_rowsets_when_load_txn_begin, "false");
-DEFINE_mBool(enable_cloud_make_rs_visible_on_be, "false");
+DEFINE_mBool(enable_cloud_make_rs_visible_on_be, "true");
DEFINE_mInt32(file_handles_deplenish_frequency_times, "3");
// clang-format off
diff --git a/be/test/agent/task_worker_pool_test.cpp
b/be/test/agent/task_worker_pool_test.cpp
index b8b2de0fc4b..8323ad1bd66 100644
--- a/be/test/agent/task_worker_pool_test.cpp
+++ b/be/test/agent/task_worker_pool_test.cpp
@@ -27,10 +27,13 @@
#include "agent/agent_server.h"
#include "cloud/cloud_storage_engine.h"
+#include "common/config.h"
#include "runtime/cluster_info.h"
#include "runtime/exec_env.h"
#include "storage/options.h"
#include "storage/storage_engine.h"
+#include "util/defer_op.h"
+#include "util/threadpool.h"
namespace doris {
@@ -106,6 +109,41 @@ TEST(TaskWorkerPoolTest, PreSubmitCallbackWithDedup) {
EXPECT_EQ(callback_count.load(), 1);
}
+class CloudCommittedRowsetTaskTest : public testing::TestWithParam<bool> {};
+
+TEST_P(CloudCommittedRowsetTaskTest, ReleasesSignatureAfterCallback) {
+ const bool original_enabled = config::enable_cloud_make_rs_visible_on_be;
+ Defer restore_config = [&] { config::enable_cloud_make_rs_visible_on_be =
original_enabled; };
+ config::enable_cloud_make_rs_visible_on_be = GetParam();
+
+ CloudStorageEngine engine(EngineOptions {});
+ std::atomic_int callback_count {0};
+ TaskWorkerPool workers("make_cloud_committed_rs_visible_test", 1,
[&](const auto& task) {
+ make_cloud_committed_rs_visible_callback(engine, task);
+ ++callback_count;
+ });
+
+ TAgentTaskRequest task;
+ task.__set_task_type(TTaskType::MAKE_CLOUD_COMMITTED_RS_VISIBLE);
+ task.__set_signature(GetParam() ? 5975401 : 5975400);
+ task.make_cloud_tmp_rs_visible_req.__set_txn_id(task.signature);
+ // A tablet absent from the local cache is skipped without fetching
metadata from MS.
+ task.make_cloud_tmp_rs_visible_req.__set_tablet_ids({59754});
+
+ ASSERT_TRUE(workers.submit_task(task).ok());
+ workers._thread_pool->wait();
+ ASSERT_EQ(callback_count.load(), 1);
+
+ // A completed notification must not suppress a later task with the same
signature,
+ // including when the feature was disabled before the first callback ran.
+ ASSERT_TRUE(workers.submit_task(task).ok());
+ workers._thread_pool->wait();
+ EXPECT_EQ(callback_count.load(), 2);
+}
+
+INSTANTIATE_TEST_SUITE_P(EnabledAndDisabled, CloudCommittedRowsetTaskTest,
+ testing::Values(true, false));
+
TEST(TaskWorkerPoolTest, PriorTaskWorkerPool) {
std::atomic_int normal_count {0};
std::atomic_int high_prior_count {0};
diff --git a/be/test/cloud/cloud_committed_rs_mgr_test.cpp
b/be/test/cloud/cloud_committed_rs_mgr_test.cpp
index 9df6597b8a6..791f168919e 100644
--- a/be/test/cloud/cloud_committed_rs_mgr_test.cpp
+++ b/be/test/cloud/cloud_committed_rs_mgr_test.cpp
@@ -25,6 +25,7 @@
#include "cloud/config.h"
#include "storage/rowset/rowset_meta.h"
+#include "util/defer_op.h"
namespace doris {
@@ -179,6 +180,55 @@ TEST_F(CloudCommittedRSMgrTest,
TestMarkAndCheckEmptyRowset) {
EXPECT_EQ(retrieved_expiration, txn_expiration);
}
+TEST_F(CloudCommittedRSMgrTest, TestRemoveEmptyRowset) {
+ const int64_t expiration_time = current_time_seconds() + 3600;
+ _mgr->mark_empty_rowset(1000, 2000, expiration_time);
+ _mgr->mark_empty_rowset(1000, 2001, expiration_time);
+ _mgr->mark_empty_rowset(1001, 2000, expiration_time);
+
+ _mgr->remove_committed_rowset(1000, 2000);
+ auto result = _mgr->get_committed_rowset(1000, 2000);
+ ASSERT_FALSE(result.has_value());
+ EXPECT_TRUE(result.error().is<ErrorCode::NOT_FOUND>());
+ EXPECT_TRUE(_mgr->get_committed_rowset(1000, 2001).has_value());
+ EXPECT_TRUE(_mgr->get_committed_rowset(1001, 2000).has_value());
+
+ // Repeated removal is harmless, and a consumed empty marker cannot mask a
new entry.
+ _mgr->remove_committed_rowset(1000, 2000);
+ auto rowset_meta = create_rowset_meta(2000, 1000);
+ _mgr->add_committed_rowset(1000, 2000, rowset_meta, expiration_time);
+ _mgr->remove_expired_committed_rowsets();
+ result = _mgr->get_committed_rowset(1000, 2000);
+ ASSERT_TRUE(result.has_value());
+ EXPECT_EQ(result->first, rowset_meta);
+}
+
+TEST_F(CloudCommittedRSMgrTest, TestRefreshExpiration) {
+ const auto original_min_expired_seconds =
config::tablet_txn_info_min_expired_seconds;
+ Defer restore_config = [&] {
+ config::tablet_txn_info_min_expired_seconds =
original_min_expired_seconds;
+ };
+ config::tablet_txn_info_min_expired_seconds = -100;
+ const int64_t now = current_time_seconds();
+
+ // An old expiration index entry must not remove either kind of refreshed
cache entry.
+ auto rowset_meta = create_rowset_meta(2000, 1000);
+ _mgr->add_committed_rowset(1000, 2000, rowset_meta, now - 10);
+ _mgr->mark_empty_rowset(1000, 2001, now - 10);
+ _mgr->add_committed_rowset(1000, 2000, rowset_meta, now + 3600);
+ _mgr->mark_empty_rowset(1000, 2001, now + 3600);
+ _mgr->remove_expired_committed_rowsets();
+
+ auto result = _mgr->get_committed_rowset(1000, 2000);
+ ASSERT_TRUE(result.has_value());
+ EXPECT_EQ(result->first, rowset_meta);
+ EXPECT_EQ(result->second, now + 3600);
+ result = _mgr->get_committed_rowset(1000, 2001);
+ ASSERT_TRUE(result.has_value());
+ EXPECT_EQ(result->first, nullptr);
+ EXPECT_EQ(result->second, now + 3600);
+}
+
TEST_F(CloudCommittedRSMgrTest, TestEmptyRowsetExpiration) {
// Save original config value
int32_t original_min_expired_seconds =
config::tablet_txn_info_min_expired_seconds;
diff --git a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
index 6c201e1ad4e..69a8840b7e9 100644
--- a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
+++ b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
@@ -4058,7 +4058,7 @@ public class Config extends ConfigBase {
public static long mow_get_ms_lock_retry_backoff_interval = 80;
@ConfField(mutable = true, masterOnly = true)
- public static boolean enable_notify_be_after_load_txn_commit = false;
+ public static boolean enable_notify_be_after_load_txn_commit = true;
// ATTN: DONOT add any config not related to cloud mode here
// ATTN: DONOT add any config not related to cloud mode here
diff --git a/regression-test/pipeline/cloud_p0/conf/be_custom.conf
b/regression-test/pipeline/cloud_p0/conf/be_custom.conf
index b2aef18c55c..3d5ef2e3ad1 100644
--- a/regression-test/pipeline/cloud_p0/conf/be_custom.conf
+++ b/regression-test/pipeline/cloud_p0/conf/be_custom.conf
@@ -59,8 +59,6 @@ enable_segment_rows_consistency_check=true
enable_segment_rows_check_core=true
fail_when_segment_rows_not_in_rowset_meta=true
-enable_cloud_make_rs_visible_on_be=true
-cloud_mow_sync_rowsets_when_load_txn_begin=false
# enable to use python udf
enable_python_udf_support=true
# The CI image supplies a Python venv; conda is not installed in this runtime.
diff --git a/regression-test/pipeline/cloud_p0/conf/fe_custom.conf
b/regression-test/pipeline/cloud_p0/conf/fe_custom.conf
index d6c2070aa93..2e4663e7ee6 100644
--- a/regression-test/pipeline/cloud_p0/conf/fe_custom.conf
+++ b/regression-test/pipeline/cloud_p0/conf/fe_custom.conf
@@ -48,5 +48,4 @@ enable_advance_next_id = true
check_table_lock_leaky = true
enable_outfile_to_local=true
-enable_notify_be_after_load_txn_commit=true
-max_bucket_num_per_partition=512
\ No newline at end of file
+max_bucket_num_per_partition=512
diff --git a/regression-test/pipeline/cloud_p1/conf/be_custom.conf
b/regression-test/pipeline/cloud_p1/conf/be_custom.conf
index 0a9503ae878..8606b8aa199 100644
--- a/regression-test/pipeline/cloud_p1/conf/be_custom.conf
+++ b/regression-test/pipeline/cloud_p1/conf/be_custom.conf
@@ -41,8 +41,6 @@ enable_prefill_all_dbm_agg_cache_after_compaction=true
enable_segment_rows_consistency_check=true
enable_segment_rows_check_core=true
-enable_cloud_make_rs_visible_on_be=true
-cloud_mow_sync_rowsets_when_load_txn_begin=false
# enable to use python udf
enable_python_udf_support=true
python_env_mode=conda
diff --git a/regression-test/pipeline/cloud_p1/conf/fe_custom.conf
b/regression-test/pipeline/cloud_p1/conf/fe_custom.conf
index 157f3a07a09..b91a4ed6d38 100644
--- a/regression-test/pipeline/cloud_p1/conf/fe_custom.conf
+++ b/regression-test/pipeline/cloud_p1/conf/fe_custom.conf
@@ -37,5 +37,3 @@ enable_advance_next_id = true
arrow_flight_sql_port = 8081
enable_job_schedule_second_for_test = true
-
-enable_notify_be_after_load_txn_commit=true
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]