bobhan1 opened a new pull request, #70760: URL: https://github.com/apache/doris/pull/70760
### Background Cloud merge-on-write loads acquire a table-level lock in Meta Service before publishing delete bitmaps. Compaction and schema change also acquire locks through Meta Service. Cloud clusters currently select the lock protocol by instance using the Meta Service allowlist. ### What problem does this PR solve? Issue Number: None Related PR: #50173 (introduced the instance allowlist) PR Type: Enhancement Problem Summary: The allowlist makes enabling V2 an instance-wide operation. Operators have to coordinate a protocol switch across clients and Meta Service while existing tables are active. Store the protocol choice when a table is created so the cluster can use V2 for new tables while serving existing tables with their current mode. ### Details - **Change map:** The diff has 200 production lines and 228 test lines, including nine lines of generated SQL results. Table creation and tablet metadata carry the immutable setting. Lock RPCs and existing tablet job records carry the setting to Meta Service. The tests cover selection, cleanup, and SQL behavior. - **Before and after:** Meta Service previously selected V1 or V2 from its instance configuration. It now uses the table setting when supplied. FE sends it for load locks and BE sends it for compaction, schema change, and locked bitmap writes. - **Configs and defaults:** Cloud MoW `CREATE TABLE` defaults `enable_delete_bitmap_lock_v2` to `true`. Set `"enable_delete_bitmap_lock_v2" = "false"` at creation to use V1. `ALTER TABLE` rejects changes to the property. The existing `delete_bitmap_lock_v2_white_list` is the fallback for tables without the setting. - **Impact on existing users:** Upgraded tables with no stored setting keep their allowlist-selected mode. New partitions, shadow tablets, and restored tablets use the destination table's setting. Roll out support to FE, BE, Meta Service and Recycler before creating tables that use the property. - **Behavior change:** New cloud MoW tables use V2 by default. ### Testing - Local ASAN Cloud UT passed the two new `*TableDeleteBitmapLockVersion*` tests for explicit V1 and V2, legacy fallback, locked update and release, and V2 lock cleanup through the recorded job. - Six existing `*DeleteBitmapUpdateLock*` Cloud UTs passed locally with ASAN. - `CloudInternalCatalogTest.testCreateTabletsCarriesTableDeleteBitmapLockVersion` passed locally for true, false, and absent properties in the create-tablets RPC. - `test_table_delete_bitmap_lock_version` passed on a local cloud cluster with FoundationDB and S3. It covers the new default, an explicit V1 table, invalid DDL, and MoW overwrites in both tables. - Manual full compaction on each table merged [2-2] and [3-3]. Meta Service used the table's protocol for lock acquisition and job commit, and both queries returned the expected rows after compaction. ### Notes for Reviewers - The mode comes from existing tablet metadata on BE. Meta Service selects the protocol from the RPC and resolves it from the existing job record when finishing a job. Snapshot restore takes the mode from the destination tablet because the source header may come from a table with a different setting. ### Release note New cloud Merge-on-Write tables use delete bitmap lock V2 by default. To use V1 for a new table, set `enable_delete_bitmap_lock_v2` to `false` in the CREATE TABLE properties. Tables without the property continue to use the Meta Service instance allowlist. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: Yes (document the property and cloud upgrade behavior in the CREATE TABLE reference) - Pre-submission status: Ready ### AI Disclosure - Tool: Codex (GPT-6) - Generated: Implementation, tests, and PR description - Human review: Pending at submission -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
