dxyou0830 opened a new pull request, #70585:
URL: https://github.com/apache/doris/pull/70585

   ### Background
   
   Snapshot conversion rewrites rowset metadata when cloning replicas or 
restoring a snapshot into another tablet. Multiple inverted indexes can share a 
column, index type and suffix while having different names and properties, such 
as an exact index and a lowercase-normalizer index.
   
   ### What problem does this PR solve?
   
   Issue Number: close #70583
   Related PR: None
   PR Type: Bugfix
   
   Problem Summary:
   
   Both local and cloud snapshot conversion look up the first target index for 
a column/type/suffix tuple. This can rewrite two distinct inverted indexes to 
the same ID. An incremental clone into an existing tablet can therefore leave 
duplicate index IDs in copied rowset schemas, causing subsequent compaction to 
fail with `[E-6010] IndexFileWriter::open attempted to insert a duplicate dir`. 
Preserve source IDs for same-tablet replica clones, and match complete index 
definitions where restore requires remapping.
   
   ### Details
   
   - **Frequency:** Reproducible when a column has multiple inverted indexes 
with the same type and suffix and snapshot conversion has an existing target 
schema. Reversing target index order changes which ID survives in the old 
implementation.
   - **Root cause:** `TabletSchema::get_index` returns the first matching 
column/type/suffix entry; that tuple does not uniquely identify an index 
definition.
   - **Why existing tests missed it:** The local conversion test uses one 
bitmap index. The cloud remapping test uses indexes of different types rather 
than two inverted indexes on the same column.
   - **Fix:** Same-tablet local clones retain the source index IDs, including 
historical rowsets. Local restore and cloud remapping use the existing 
`TabletIndex::is_same_except_id` comparison, which includes name, type, column 
IDs, suffix and properties. Existing behavior for an absent target schema or 
unmatched definition is retained.
   - **Other paths and branches:** Both `SnapshotManager` and 
`CloudSnapshotMgr` are covered. The local test checks tablet, visible-rowset 
and stale-rowset schemas. This PR targets master; no release-branch changes are 
included.
   - **Behavior change:** Distinct index definitions remain distinct after 
conversion. No configuration, protocol or storage-format changes are 
introduced. Already-corrupted rowset metadata is not repaired by this change.
   
   ### Testing
   
   - Added `SnapshotManagerTest.ConvertDualInvertedIndexesIntoExistingTablet`: 
create an existing target tablet, convert missing-version snapshot metadata 
with exact and normalizer indexes, and check distinct IDs in tablet, visible 
and stale schemas for clone and restore. Target index order is reversed.
   - Added `RenameDualInvertedIndexesByDefinition` to both snapshot-manager 
suites: changed target IDs, reversed order, mismatched name/properties/suffix, 
and absent target schema.
   - Local static checks: clang-format 16, build-hygiene checks and `git diff 
--check`.
   - Separate isolated validation: synthetic writes with incremental clone into 
an existing target reproduced E-6010 on an unpatched 4.1.4 baseline; a 4.1.5 
build with the local fix retained unique index IDs and completed 28 
compactions. Exact and case-insensitive query results, Stream Load and Routine 
Load checks passed, including an older source replica cloning into a patched 
target.
   - The new master unit tests and Cloud path have not been executed locally. 
The available build cache and third-party dependencies are for 4.1.5 and cannot 
be reused as a master incremental build. Master compilation, unit tests and 
clang-tidy are pending upstream CI.
   
   ### Release note
   
   Fix snapshot index-ID remapping that could collapse multiple inverted 
indexes on one column and cause compaction failures after replica clone or 
restore.
   
   ### Check List (For Author)
   
   - Behavior changed: Yes (see Details).
   - Does this need documentation: No; this restores existing clone/restore 
behavior.
   - Pre-submission status: Partially verified: master compilation, unit tests 
and clang-tidy are pending CI; isolated local-path validation was performed on 
4.1.5.
   
   ### AI Disclosure
   
   - Tool: OpenAI Codex (GPT-6) for code, tests and description; reviewed by 
Anthropic Claude.
   - Generated: Code, tests and this description.
   - Human review: Not yet performed; requesting maintainer review.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to