Doris-Breakwater commented on issue #69410: URL: https://github.com/apache/doris/issues/69410#issuecomment-6321067068
Breakwater-GitHub-Analysis-Slot: slot_c1e92b02994b This is a credible schema-change/publish **liveness defect**, with a concrete dependency cycle supported by the source. Suggested triage: high priority, FE transactions/schema change + BE MoW publish. The issue is currently open with no labels. The reported impact is partition-wide visibility blockage for committed writes, rather than demonstrated permanent data loss. I inspected the local source at **15eb3f4e432b1d4a39f89457956135da80b90940** and **917b20be83878e98f94e3ebf7efccc80346e3e50** (the supplied 4.1.5 build prefix). Both contain the relevant behavior. This is source-level verification against the supplied traces; I have not independently run the SQL reproduction or validated a fix. **Verified mechanisms and their implications** 1. On master, `EnginePublishVersionTask` allows an initial discontinuous publish on a `TABLET_NOTREADY` MoW tablet, but rejects a subsequent forward gap when `max_version > 1` and the continuous prefix does not reach `max_version`. Thus B at v3 can publish while historical conversion is pending; after A1 receives v4 only on the original tablet, C at v5 hits the rejection with shadow `max_version = 3`. This is more specific than a blanket prohibition of gaps on every NOTREADY publish. See [the exact BE predicate](https://github.com/apache/doris/blob/15eb3f4e432b1d4a39f89457956135da80b90940/be/src/storage/task/engine_publish_version_task.cpp#L205-L240). 2. FE's `checkReplicaContinuousVersionSucc` treats an ALTER replica as successful without a successful publish for transactions below the watershed, or when `publish_version_check_alter_replica=false`. C is above the watershed and actually loaded the shadow index, so the default setting requires its shadow publish to succeed. With one replica, the shadow tablet's zero successes fail its own quorum; success on the original tablet cannot substitute for it. See [replica classification](https://github.com/apache/doris/blob/15eb3f4e432b1d4a39f89457956135da80b90940/fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java#L1642-L1680) and [quorum evaluation](https://github.com/apache/doris/blob/15eb3f4e432b1d4a39f89457956135da80b90940/fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java#L3175-L3233). 3. Partition visibility requires the preceding version to be visible. Consequently A2 at v6 cannot become VISIBLE before C at v5, even though A2's transaction ID is below the watershed. `runWaitingTxnJob` waits for unfinished earlier transactions, including COMMITTED A2, before sending historical conversion tasks. See [visibility ordering](https://github.com/apache/doris/blob/15eb3f4e432b1d4a39f89457956135da80b90940/fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java#L1506-L1538), [conversion gate](https://github.com/apache/doris/blob/15eb3f4e432b1d4a39f89457956135da80b90940/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeJobV2.java#L471-L490), and [unfinished-transaction selection](https://github.com/apache/doris/blob/15eb3f4e432b1d4a39f89457956135da80b90940/fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java#L2692-L2713). The resulting cycle is: `C shadow publish → missing history → schema-change conversion → A2 VISIBLE → C VISIBLE → C shadow publish`. A1 has already become VISIBLE; **A2** is the earlier transaction that closes the cycle. Transaction-ID order and assigned partition-version order differ here. This also explains why removing ALTER or using MoR avoids this particular MoW check. The older safeguard in [#26428](https://github.com/apache/doris/pull/26428) does not cover C: it exempts earlier transactions, whereas C is above the watershed. No FE restart is needed to form this cycle. **Recovery and fix direction** - If abandoning this schema change is acceptable, the reported `CANCEL ALTER TABLE COLUMN FROM scdl.t` is consistent with `cancelInternal`, which removes shadow indexes. Verify the job is CANCELLED and the existing committed transaction IDs become VISIBLE before resubmitting work. A COMMITTED response or visibility timeout is not an aborted write; avoid blindly retrying the same load. - The reported temporary `ADMIN SET FRONTEND CONFIG ('publish_version_check_alter_replica' = 'false')` also matches the FE mechanism. Its [configuration description](https://github.com/apache/doris/blob/15eb3f4e432b1d4a39f89457956135da80b90940/fe/fe-common/src/main/java/org/apache/doris/common/Config.java#L518-L521) explicitly reserves it for deadlock recovery. If used, monitor both transaction visibility and the ALTER outcome, then restore `true`; do not make disabling this check the permanent fix. The restart case shows that releasing transactions does not guarantee successful ALTER completion. - Prioritize a regression for the exact A1/A2/B/C ordering and review the interaction between NOTREADY publish continuity and FE's conversion gate. Any change must preserve normal RUNNING-tablet MoW continuity and correct delete-bitmap reconstruction. [Historical conversion and bitmap catch-up](https://github.com/apache/doris/blob/15eb3f4e432b1d4a39f89457956135da80b90940/be/src/storage/schema_change/schema_change.cpp#L1615-L1676) are part of the correctness contract. Test overlapping keys/updates as well as the six distinct keys, both ALTER forms, restart/replay, cancellation, and multiple replicas. Simply ignoring shadow failures globally is not a validated fix. - Clone/retry settings do not supply an old transaction's absent shadow rowset in the reported single-replica case. Increasing publish waits likewise does not remove the dependency cycle. **Evidence still needed for runtime confirmation** The report already supplies enough detail to begin a regression; another generic reproduction description or query profile is unnecessary. For one stuck run, please attach synchronized FE transaction/publish/schema-change logs and BE publish logs covering B through A2, along with: - The original/shadow tablet and replica mapping, partition visible/committed versions, shadow rowset version ranges, and tablet state. These should establish B=v3, no shadow v4, C=v5 failing on that shadow, and A2=v6, including parent/subtransaction IDs for explicit transactions. - For the restart-only `TQueryGlobals(now_string:null)` failure, the complete FE task-submission/serialization exception and replay timeline, plus confirmation whether it reproduces without restarting. That is a separate follow-up failure after the visibility cycle is released; its precise cause is not established by the supplied excerpt. No code changes or runtime recovery actions were performed in this investigation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
