This is an automated email from the ASF dual-hosted git repository.
Yukang-Lian pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 0ba9b361c79 [fix](cloud) let a restarted schema change replace its
stale STOP_TOKEN (#68398)
0ba9b361c79 is described below
commit 0ba9b361c79d0bf0bd2cb826e5786ed7e8c06b7f
Author: Jamie <[email protected]>
AuthorDate: Thu Oct 22 14:53:42 2026 +0800
[fix](cloud) let a restarted schema change replace its stale STOP_TOKEN
(#68398)
### What problem does this PR solve?
Related PR: #48399 (introduced the schema change `STOP_TOKEN`), #49275
(unregisters the token when `commit_tablet_job` fails), #66082 (a load
must not preempt the schema change delete bitmap lock)
Problem Summary:
A heavy schema change on a cloud MOW table gets CANCELLED when its BE
restarts while the delete bitmap is being recalculated, and loads on the
table fail with delete bitmap lock conflicts for a while afterwards.
`CloudSchemaChangeJob::_process_delete_bitmap` registers a compaction
`STOP_TOKEN` on the new tablet before it recalculates the delete bitmap.
When the BE dies at that point, nothing unregisters the token: it stays
in the meta service until its lease expires
(`lease_compaction_interval_seconds * 4` = 80s), and the dead run's
initiator stays in the table's schema change delete bitmap lock
(`lock_id=-2`).
FE re-sends the ALTER task a couple of seconds after the BE comes back.
The schema change job record survives that restart on purpose
(`start_schema_change_job` takes the "same job restarts" path because
the job id and the BE-address initiator are unchanged), but the new run
uses a fresh `delete_bitmap_lock_initiator`, and `start_compaction_job`
rejects its `STOP_TOKEN` because the stale one is still there:
```
failed to start tablet job: compactions are not allowed on tablet_id=...
currently, blocked by schema change job delete_bitmap_initiator=...
```
The BE reports that as `INTERNAL_ERROR`, which
`AlterJobV2.getRetryTimes` does not retry, so FE cancels the whole
schema change job. The new run also cannot release the stale lock
(`failed to remove delete bitmap update lock: lock initiator not
exist`), so until it expires (`delete_bitmap_lock_expiration_seconds`)
loads fail with `Failed to get delete bitmap lock due to conflict`: a
load may force-take a load/compaction lock but never a schema change
lock (#66082).
The same check ordering also breaks idempotency: a retried `STOP_TOKEN`
registration of the same run hits the `STOP_TOKEN` conflict check before
the same-id check and gets `JOB_TABLET_BUSY`.
### Changes
`start_compaction_job` no longer rejects an incoming `STOP_TOKEN`
because of an existing `STOP_TOKEN`. The incoming token replaces any
stale token whose `delete_bitmap_lock_initiator` differs, and the schema
change delete bitmap lock held by that stale initiator is released in
the same transaction (`remove_delete_bitmap_update_lock` with
`SCHEMA_CHANGE_DELETE_BITMAP_LOCK_ID`, respecting the instance's v1/v2
lock version, which `start_tablet_job` now passes in). FE guarantees at
most one schema change job per tablet, so the incoming token always
belongs to the live run. Regular compactions are still blocked by a live
token, and the BE/FE sides are unchanged.
New unit test `MetaServiceJobTest.StopTokenReplacesStaleStopToken`: a
stale token plus a `-2` lock held by the dead initiator; a regular
compaction is still rejected; the new token is accepted and the stale
token and lock are gone; re-registering the same token is idempotent;
the live token still blocks compactions.
### Testing
- `./run-cloud-ut.sh --run
--filter='meta_service_test:MetaServiceJobTest.*'` passes, including the
new test.
- Reproduced and verified end to end on a single FE/BE cloud cluster
with a debug point that blocks the schema change right after it acquires
the delete bitmap lock, then restarting the BE: without this change the
re-sent ALTER task is rejected and the job is CANCELLED every time; with
it the meta service logs `replace stale STOP_TOKEN ...` followed by
`remove delete bitmap lock ... lock_id=-2 initiator=<stale>`, the same
job finishes within a few seconds of the restart, and loads on the table
are not blocked.
### Release note
Cloud MOW: a heavy schema change no longer gets CANCELLED when its BE
restarts while the delete bitmap is being recalculated, and loads on the
table are no longer blocked by the dead run's schema change lock.
---
cloud/src/meta-service/meta_service_job.cpp | 44 +++++++--
cloud/test/meta_service_job_test.cpp | 134 ++++++++++++++++++++++++++++
2 files changed, 169 insertions(+), 9 deletions(-)
diff --git a/cloud/src/meta-service/meta_service_job.cpp
b/cloud/src/meta-service/meta_service_job.cpp
index 6818968c821..34dda7ec920 100644
--- a/cloud/src/meta-service/meta_service_job.cpp
+++ b/cloud/src/meta-service/meta_service_job.cpp
@@ -146,11 +146,16 @@ static inline bool
may_conflict_by_type(TabletCompactionJobPB::CompactionType a,
return is_rowset_compaction_family(a) && is_rowset_compaction_family(b);
}
+static void remove_delete_bitmap_update_lock(std::unique_ptr<Transaction>& txn,
+ const std::string& instance_id,
int64_t table_id,
+ int64_t tablet_id, int64_t
lock_id,
+ int64_t lock_initiator,
std::string use_version);
+
void start_compaction_job(MetaServiceCode& code, std::string& msg,
std::stringstream& ss,
std::unique_ptr<Transaction>& txn, const
StartTabletJobRequest* request,
StartTabletJobResponse* response, std::string&
instance_id,
- bool& need_commit, bool is_versioned_read,
- ResourceManager* resource_mgr) {
+ bool& need_commit, bool is_versioned_read,
ResourceManager* resource_mgr,
+ const std::string& use_version) {
auto& compaction = request->job().compaction(0);
if (!compaction.has_id() || compaction.id().empty()) {
code = MetaServiceCode::INVALID_ARGUMENT;
@@ -274,12 +279,12 @@ void start_compaction_job(MetaServiceCode& code,
std::string& msg, std::stringst
}), compactions.end());
// clang-format on
// Check conflict job
- if (std::ranges::any_of(compactions, [](const auto& c) {
- return c.type() == TabletCompactionJobPB::STOP_TOKEN;
- })) {
- auto it = std::ranges::find_if(compactions, [](const auto& c) {
- return c.type() == TabletCompactionJobPB::STOP_TOKEN;
- });
+ auto is_stop_token = [](const auto& c) {
+ return c.type() == TabletCompactionJobPB::STOP_TOKEN;
+ };
+ if (compaction.type() != TabletCompactionJobPB::STOP_TOKEN &&
+ std::ranges::any_of(compactions, is_stop_token)) {
+ auto it = std::ranges::find_if(compactions, is_stop_token);
msg = fmt::format(
"compactions are not allowed on tablet_id={} currently,
blocked by schema "
"change job delete_bitmap_initiator={}",
@@ -293,6 +298,25 @@ void start_compaction_job(MetaServiceCode& code,
std::string& msg, std::stringst
// compaction operations and canceling other types of compaction.
compactions.Clear();
} else if (compaction.type() == TabletCompactionJobPB::STOP_TOKEN) {
+ // A STOP_TOKEN outlives the schema change run that registered it
when its BE dies
+ // before unregistering it: nothing renews or removes the token
until its lease
+ // expires (lease_compaction_interval_seconds * 4). FE guarantees
at most one schema
+ // change job per tablet and every run uses a fresh
delete_bitmap_lock_initiator, so
+ // an incoming STOP_TOKEN always belongs to the live run. Replace
the stale token and
+ // release the schema change delete bitmap lock its dead owner
still holds; otherwise
+ // the re-sent ALTER task is rejected here and FE cancels the
whole job.
+ for (const auto& c : compactions) {
+ if (!is_stop_token(c) || !c.has_delete_bitmap_lock_initiator()
||
+ c.delete_bitmap_lock_initiator() ==
compaction.delete_bitmap_lock_initiator()) {
+ continue;
+ }
+ INSTANCE_LOG(INFO)
+ << "replace stale STOP_TOKEN, tablet_id=" << tablet_id
+ << " new_job_id=" << compaction.id() << " stale_job="
<< proto_to_json(c);
+ remove_delete_bitmap_update_lock(txn, instance_id, table_id,
tablet_id,
+
SCHEMA_CHANGE_DELETE_BITMAP_LOCK_ID,
+
c.delete_bitmap_lock_initiator(), use_version);
+ }
// fail all existing compactions
compactions.Clear();
} else if ((!compaction.has_check_input_versions_range() &&
@@ -604,8 +628,10 @@ void
MetaServiceImpl::start_tablet_job(::google::protobuf::RpcController* contro
};
if (!request->job().compaction().empty()) {
+ std::string use_version =
+
delete_bitmap_lock_white_list_->get_delete_bitmap_lock_version(instance_id);
start_compaction_job(code, msg, ss, txn, request, response,
instance_id, need_commit,
- is_versioned_read, resource_mgr_.get());
+ is_versioned_read, resource_mgr_.get(),
use_version);
return;
}
diff --git a/cloud/test/meta_service_job_test.cpp
b/cloud/test/meta_service_job_test.cpp
index 2e184c2ee9d..87a07325294 100644
--- a/cloud/test/meta_service_job_test.cpp
+++ b/cloud/test/meta_service_job_test.cpp
@@ -1788,6 +1788,140 @@ TEST(MetaServiceJobTest,
StopTokenSkipsStaleTabletCacheCheck) {
}
}
+// A STOP_TOKEN left behind by a schema change run whose BE
+// died before unregistering it stays valid until its lease expires. The
re-sent ALTER task
+// registers a new STOP_TOKEN with a fresh delete_bitmap_lock_initiator; it
must replace the
+// stale token and release the schema change delete bitmap lock the dead run
still holds,
+// instead of being rejected with JOB_TABLET_BUSY (which makes FE cancel the
whole job).
+TEST(MetaServiceJobTest, StopTokenReplacesStaleStopToken) {
+ auto meta_service = get_meta_service();
+
+ auto sp = SyncPoint::get_instance();
+ DORIS_CLOUD_DEFER {
+ SyncPoint::get_instance()->clear_all_call_backs();
+ };
+ sp->set_call_back("get_instance_id", [&](auto&& args) {
+ auto* ret = try_any_cast_ret<std::string>(args);
+ ret->first = instance_id;
+ ret->second = true;
+ });
+ sp->enable_processing();
+
+ int64_t table_id = 1, index_id = 2, partition_id = 3, tablet_id = 102;
+ constexpr int64_t stale_initiator = 111;
+ constexpr int64_t new_initiator = 222;
+
+ {
+ auto index_key = meta_tablet_idx_key({instance_id, tablet_id});
+ TabletIndexPB idx_pb;
+ idx_pb.set_table_id(table_id);
+ idx_pb.set_index_id(index_id);
+ idx_pb.set_partition_id(partition_id);
+ idx_pb.set_tablet_id(tablet_id);
+ std::string stats_key =
+ stats_tablet_key({instance_id, table_id, index_id,
partition_id, tablet_id});
+ TabletStatsPB stats;
+ stats.set_base_compaction_cnt(0);
+ stats.set_cumulative_compaction_cnt(0);
+ std::unique_ptr<Transaction> txn;
+ ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn),
TxnErrorCode::TXN_OK);
+ txn->put(index_key, idx_pb.SerializeAsString());
+ txn->put(stats_key, stats.SerializeAsString());
+ ASSERT_EQ(txn->commit(), TxnErrorCode::TXN_OK);
+ }
+
+ auto register_stop_token = [&](const std::string& job_id, int64_t
initiator,
+ StartTabletJobResponse& res) {
+ brpc::Controller cntl;
+ StartTabletJobRequest req;
+ req.mutable_job()->mutable_idx()->set_tablet_id(tablet_id);
+ auto* compaction = req.mutable_job()->add_compaction();
+ compaction->set_id(job_id);
+ compaction->set_initiator("ip:port");
+ compaction->set_type(TabletCompactionJobPB::STOP_TOKEN);
+ compaction->set_base_compaction_cnt(0);
+ compaction->set_cumulative_compaction_cnt(0);
+ long now = time(nullptr);
+ compaction->set_expiration(now + 12);
+ compaction->set_lease(now + 80);
+ compaction->set_delete_bitmap_lock_initiator(initiator);
+ meta_service->start_tablet_job(&cntl, &req, &res, nullptr);
+ };
+ auto read_tablet_job = [&]() {
+ TabletJobInfoPB job_pb;
+ std::unique_ptr<Transaction> txn;
+ EXPECT_EQ(meta_service->txn_kv()->create_txn(&txn),
TxnErrorCode::TXN_OK);
+ std::string job_val;
+ EXPECT_EQ(
+ txn->get(job_tablet_key({instance_id, table_id, index_id,
partition_id, tablet_id}),
+ &job_val),
+ TxnErrorCode::TXN_OK);
+ EXPECT_TRUE(job_pb.ParseFromString(job_val));
+ return job_pb;
+ };
+
+ // The dead run registered its STOP_TOKEN and still holds the schema
change lock.
+ {
+ StartTabletJobResponse res;
+ register_stop_token("stale_stop_token", stale_initiator, res);
+ ASSERT_EQ(res.status().code(), MetaServiceCode::OK) <<
res.status().msg();
+ }
+ {
+ DeleteBitmapUpdateLockPB lock_info;
+ lock_info.set_lock_id(SCHEMA_CHANGE_DELETE_BITMAP_LOCK_ID);
+ lock_info.set_expiration(time(nullptr) + 60);
+ lock_info.add_initiators(stale_initiator);
+ std::unique_ptr<Transaction> txn;
+ ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn),
TxnErrorCode::TXN_OK);
+ txn->put(meta_delete_bitmap_update_lock_key({instance_id, table_id,
-1}),
+ lock_info.SerializeAsString());
+ ASSERT_EQ(txn->commit(), TxnErrorCode::TXN_OK);
+ }
+
+ // A regular compaction is still blocked by the unexpired STOP_TOKEN.
+ {
+ StartTabletJobResponse res;
+ start_compaction_job(meta_service.get(), tablet_id, "cumu_job_1",
"ip:port",
+ /*base_cnt=*/0, /*cumu_cnt=*/0,
TabletCompactionJobPB::CUMULATIVE,
+ res);
+ ASSERT_EQ(res.status().code(), MetaServiceCode::JOB_TABLET_BUSY);
+ }
+
+ // The re-sent ALTER task registers a new STOP_TOKEN: it replaces the
stale one and the
+ // dead run's schema change lock is released.
+ {
+ StartTabletJobResponse res;
+ register_stop_token("new_stop_token", new_initiator, res);
+ ASSERT_EQ(res.status().code(), MetaServiceCode::OK)
+ << "new STOP_TOKEN must replace the stale one; got: " <<
res.status().msg();
+ auto job_pb = read_tablet_job();
+ ASSERT_EQ(job_pb.compaction_size(), 1);
+ EXPECT_EQ(job_pb.compaction(0).id(), "new_stop_token");
+ EXPECT_EQ(job_pb.compaction(0).delete_bitmap_lock_initiator(),
new_initiator);
+ check_delete_bitmap_lock(meta_service.get(), instance_id, table_id,
+ SCHEMA_CHANGE_DELETE_BITMAP_LOCK_ID,
/*exist=*/false);
+ }
+
+ // Re-registering the same STOP_TOKEN (RPC retry) is idempotent.
+ {
+ StartTabletJobResponse res;
+ register_stop_token("new_stop_token", new_initiator, res);
+ ASSERT_EQ(res.status().code(), MetaServiceCode::OK) <<
res.status().msg();
+ auto job_pb = read_tablet_job();
+ ASSERT_EQ(job_pb.compaction_size(), 1);
+ EXPECT_EQ(job_pb.compaction(0).id(), "new_stop_token");
+ }
+
+ // The live token keeps blocking regular compactions.
+ {
+ StartTabletJobResponse res;
+ start_compaction_job(meta_service.get(), tablet_id, "cumu_job_2",
"ip:port",
+ /*base_cnt=*/0, /*cumu_cnt=*/0,
TabletCompactionJobPB::CUMULATIVE,
+ res);
+ ASSERT_EQ(res.status().code(), MetaServiceCode::JOB_TABLET_BUSY);
+ }
+}
+
TEST(MetaServiceJobTest, DeleteBitmapUpdateLockCompatibilityTest) {
auto meta_service = get_meta_service();
auto sp = SyncPoint::get_instance();
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
