Doris-Breakwater commented on issue #68679: URL: https://github.com/apache/doris/issues/68679#issuecomment-5907593279
Breakwater-GitHub-Analysis-Slot: slot_9c2dc24ef6e2 **Initial assessment (read-only source review): high-priority correctness risk for stream-consuming `INSERT OVERWRITE`; no production occurrence or remote/cloud fault-injection result is established by this issue.** I checked `apache/doris` master at [`7d231485588`](https://github.com/apache/doris/commit/7d2314855883bab61255d77c6c18d8d535357a68) (2026-09-30 08:25 UTC, just before this issue). The issue currently has no labels. **Confirmed from code** - The inner insert attaches stream updates to its transaction; the local transaction applies them after committing and on replay ([insert setup](https://github.com/apache/doris/blob/7d2314855883bab61255d77c6c18d8d535357a68/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertIntoTableCommand.java#L348-L369), [commit/replay](https://github.com/apache/doris/blob/7d2314855883bab61255d77c6c18d8d535357a68/fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java#L2410-L2422)). `InsertOverwriteTableCommand` then swaps temporary partitions separately and calls `taskFail` on a swap exception ([overwrite sequence](https://github.com/apache/doris/blob/7d2314855883bab61255d77c6c18d8d535357a68/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertOverwriteTableCommand.java#L288-L312)); the swap has its own edit-log record ([replacement log](https://github.com/apache/doris/blob/7d2314855883bab61255d77c6c18d8d5 35357a68/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java#L7079-L7115)). On master transfer, outstanding tasks are failed and their temporary partitions are dropped ([recovery](https://github.com/apache/doris/blob/7d2314855883bab61255d77c6c18d8d535357a68/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java#L1889-L1895), [cleanup](https://github.com/apache/doris/blob/7d2314855883bab61255d77c6c18d8d535357a68/fe/fe-core/src/main/java/org/apache/doris/insertoverwrite/InsertOverwriteManager.java#L229-L260)). Thus a committed offset can outlive an unpublished or dropped temporary partition. A swap exception *does* surface an error for that attempt; the potentially silent loss is on a later read from the advanced offset. - A lost remote commit reply can leave the caller unable to distinguish a committed transaction from a failed one: the RPC has finite retries, and the caller's failure path aborts best-effort while overwrite cleanup can drop the remote temporary partitions ([retry](https://github.com/apache/doris/blob/7d2314855883bab61255d77c6c18d8d535357a68/fe/fe-core/src/main/java/org/apache/doris/datasource/doris/FeServiceClient.java#L201-L244), [commit/abort](https://github.com/apache/doris/blob/7d2314855883bab61255d77c6c18d8d535357a68/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/RemoteOlapInsertExecutor.java#L181-L255)). This is a conditional loss window, not proof that the reported remote case has occurred. In particular, the remote commit request does not carry stream updates, so the claimed remote *offset* impact needs a separate concrete reproduction. - Cloud commit requests include stream updates ([request](https://github.com/apache/doris/blob/7d2314855883bab61255d77c6c18d8d535357a68/fe/fe-core/src/main/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgr.java#L737-L756)). The meta-service maps exhausted `KV_TXN_MAYBE_COMMITTED` retries to `KV_TXN_COMMIT_ERR` ([retry boundary](https://github.com/apache/doris/blob/7d2314855883bab61255d77c6c18d8d535357a68/cloud/src/meta-service/meta_service.h#L1129-L1142)); FE treats that response as an error ([FE response handling](https://github.com/apache/doris/blob/7d2314855883bab61255d77c6c18d8d535357a68/fe/fe-core/src/main/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgr.java#L944-L978)). Whether any particular transaction actually committed remains unknown until checked at the owner. **Important scope correction:** current master already journals an IVM partition rebuild requirement before that refresh reads ([MTMV task](https://github.com/apache/doris/blob/7d2314855883bab61255d77c6c18d8d535357a68/fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVTask.java#L986-L1015)). The checked-in [failure-between-halves regression](https://github.com/apache/doris/blob/7d2314855883bab61255d77c6c18d8d535357a68/regression-test/suites/mtmv_p0/ivm/test_ivm_overwrite_failure_between_the_halves.groovy#L119-L148) expects the following refresh to rebuild the partition and recover the row. That limits the stated IVM impact on this master revision; it does not close the direct stream-overwrite gap. PR [#68662](https://github.com/apache/doris/pull/68662) is open, so its cancellation changes should not yet be assumed present on master. The remote lock-wait cancellation/success scenario also needs its own deterministic test. **Requested evidence / next steps:** Please provide an exact build SHA and mode (local, remote, or cloud), minimal table/stream SQL and partition layout, transaction ID, overwrite task ID, affected partition IDs, stream offset before/after, and row counts before/after retry. For failover, include FE edit-log ordering and leader-transition logs; for remote, caller/owner FE commit and abort RPC logs plus owner transaction status; for cloud, meta-service commit response/`actual_code` and authoritative transaction status. A direct stream-overwrite fault-injection test at commit-before-swap, plus lost-reply and maybe-committed tests, would establish the remaining outcomes. The fix should make publication recoverable with the committed consumption decision, and reconcile uncertain commits with the owning FE/meta-service before any temporary-partition cleanup; a local error alone cannot determine whether rollback is safe. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
