Doris-Breakwater commented on issue #68755: URL: https://github.com/apache/doris/issues/68755#issuecomment-6042291002
Breakwater-GitHub-Analysis-Slot: slot_ace030d334c0 **Assessment:** The cited upstream code confirms a coordinator-selection defect for a retained, nonzero BE ID when the transaction address differs from the registered BE host. This mechanism can explain an abandoned PREPARE transaction blocking schema change or rollup. The reported production incident and the proposed end-to-end scenario have not been independently reproduced, so this is a code-backed diagnosis under the stated conditions, not a verified deployment-wide fix. The issue is currently open with no labels, assignee, or milestone. Suggested routing: cloud transaction management, with FE heartbeat/DDL and meta-service maintainers involved. **Verified code path** I inspected upstream master `7ac0eb2fdb426595f810e94ab77b7aa2eece12d5` from the local repository and fetched the corresponding files at branch-4.1 `78aa9c996accd114a6e0f5943d7882ee1a34011a`. Both contain the reported identity predicates. 1. **Transaction identity:** Stream-load begin records the request's backend ID, connection client IP, and the backend startup time known to FE. `TxnUtil` persists these fields into the coordinator protobuf without normalizing the IP to the registered host. Consequently, hostname/IP inequality is possible without any change in backend identity. See [transaction creation](https://github.com/apache/doris/blob/7ac0eb2fdb426595f810e94ab77b7aa2eece12d5/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java#L1506-L1509) and [coordinator serialization](https://github.com/apache/doris/blob/7ac0eb2fdb426595f810e94ab77b7aa2eece12d5/fe/fe-core/src/main/java/org/apache/doris/cloud/transaction/TxnUtil.java#L327-L333). 2. **DDL failure detection:** `checkFailedTxnsByCoordinator()` finds the BE by ID and sets `offline=false`, but requires host/IP equality before evaluating restart or prolonged heartbeat loss. With the example `(123, A, T1)` versus `(123, FQDN, T2)`, the address guard is false and the method returns false even when `T2>T1`. This is directly established by the [FE predicate](https://github.com/apache/doris/blob/7ac0eb2fdb426595f810e94ab77b7aa2eece12d5/fe/fe-core/src/main/java/org/apache/doris/transaction/GlobalTransactionMgr.java#L488-L526), also present in [branch-4.1](https://github.com/apache/doris/blob/78aa9c996accd114a6e0f5943d7882ee1a34011a/fe/fe-core/src/main/java/org/apache/doris/transaction/GlobalTransactionMgr.java#L478-L516). 3. **Heartbeat cleanup:** `HeartbeatMgr` supplies `be.getId()` and `be.getHost()`. In cloud mode, cleanup calls `get_prepare_txn_by_coordinator()`, then aborts the returned transaction IDs individually. The query requires IP equality even for matching nonzero IDs, so an address mismatch yields no candidates. The separate `abort_txn_with_coordinator()` RPC also requires ID and IP equality; it should receive consistent semantics, but it is not the heartbeat cleanup RPC used by this FE path. See [heartbeat dispatch](https://github.com/apache/doris/blob/7ac0eb2fdb426595f810e94ab77b7aa2eece12d5/fe/fe-core/src/main/java/org/apache/doris/system/HeartbeatMgr.java#L195-L230), [cloud cleanup](https://github.com/apache/doris/blob/7ac0eb2fdb426595f810e94ab77b7aa2eece12d5/fe/fe-core/src/main/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgr.java#L2470-L2533), and [meta-service matching](https://github.com/apache/doris/blob/7ac0eb2fdb426595f810e94ab77b7aa2eece12d5/cloud/src/meta -service/meta_service_txn.cpp#L4642-L4815). 4. **Why DDL waits:** Schema-change and rollup helpers retrieve unfinished transactions preceding their watershed and return false while that list is nonempty. An identity miss leaves the abandoned transaction in that list. Successful abort can require another polling round before DDL advances. The three-day stream-load default exists, but actual waiting time depends on the transaction's configured timeout and expiration processing. See [schema-change helper](https://github.com/apache/doris/blob/7ac0eb2fdb426595f810e94ab77b7aa2eece12d5/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeJobV2.java#L900-L917), [rollup helper](https://github.com/apache/doris/blob/7ac0eb2fdb426595f810e94ab77b7aa2eece12d5/fe/fe-core/src/main/java/org/apache/doris/alter/RollupJobV2.java#L705-L722), and [timeout default](https://github.com/apache/doris/blob/7ac0eb2fdb426595f810e94ab77b7aa2eece12d5/fe/fe-common/src/main/java/org/apache/doris/common/Config.java#L615). **Candidate fix and safety checks** The [linked patch](https://github.com/thealtoclef/doris/commit/b472de869855d9a3a3db3ac62c5b17bbea1d18a8) addresses both identity predicates and gates the FE behavior change on cloud mode. It is a focused change with no new configuration, RPC fields, or persisted format. Stable-ID matching should be scoped to the existing instance and retained BE registration; it does not recover an old identity after re-registration with a different ID. - Preserve BE source-type and PREPARED selection checks, strict `coordinator.start_time < restart_start_time`, and FE exclusion of COMMITTED/final transactions. The abort-by-ID handler re-reads transaction state and rejects COMMITTED/VISIBLE transactions, providing an additional protection when commit races cleanup. Preserve the existing DDL retry handling on abort failure. See [abort state checks](https://github.com/apache/doris/blob/7ac0eb2fdb426595f810e94ab77b7aa2eece12d5/cloud/src/meta-service/meta_service_txn.cpp#L3880-L3918). - Explicitly document and test zero-ID compatibility. Falling back when either ID is zero broadens existing behavior: the query previously accepted a stored zero ID, but a request ID of zero did not match a stored nonzero ID. Missing ID fields must still be distinguished from explicitly supplied zero. Address fallback cannot resolve an address change when stable identity is unavailable. - **Code-supported safety concern, not runtime-reproduced:** Startup-time filtering protects restart cleanup, but BE-down cleanup calls the query with `beStartTime=0`, omitting the filter. Cleanup runs on a separate single-thread executor and does not recheck BE liveness at execution. If a down task waits in the queue, the BE recovers, and a new PREPARE load starts before that task queries meta-service, ID-only selection can include the new load despite its new address/startup time. Add a deterministic delayed-down-task/recovery test and assess liveness/epoch fencing before claiming that all new-process transactions are protected. This follows from the [executor and dispatch](https://github.com/apache/doris/blob/7ac0eb2fdb426595f810e94ab77b7aa2eece12d5/fe/fe-core/src/main/java/org/apache/doris/system/HeartbeatMgr.java#L90-L96) and the cloud cleanup linked above; it is not evidence that this race caused the reported incident. **Evidence still needed** - Exact FE, BE, and meta-service build SHAs, relevant fork differences, and effective cloud/FQDN, cleanup-switch, heartbeat-loss threshold, and stream-load timeout settings. - One correlated timeline: transaction ID/status and persisted coordinator `(source, ID, address, start_time)`; BE registration ID/host/startup time/liveness before and after rescheduling; DDL job ID, table, watershed transaction ID, and WAITING_TXN history. Redacted addresses should preserve equality/inequality relationships. - FE heartbeat/cleanup and DDL logs plus meta-service query request/status/candidate counts and abort results. Include any RPC/KV errors to distinguish an identity miss from scan or abort failure, and the actual before/after test output for the candidate patch. **Recommended next steps** 1. Port the focused identity change onto the intended upstream revision and extend native FE/meta-service tests: same ID/different address must match; different nonzero IDs/same address must not; test zero/missing IDs, FE-source records, startup times below/equal/above the cutoff, and non-PREPARED statuses. Cover both coordinator RPCs and both query scan modes. 2. Run the controlled cloud/FQDN integration scenario with heartbeat cleanup and DDL conflict cleanup enabled separately, then together. Exercise schema change and rollup, unchanged and changed pod IPs, and verify that the old transaction aborts while a new-process transaction survives and DDL finishes. Include delayed cleanup/recovery and concurrent commit/abort cases. 3. Keep the existing abort-failure recovery and scan-reliability protections when integrating/backporting. [#67661](https://github.com/apache/doris/pull/67661) and [#67761](https://github.com/apache/doris/pull/67761) / [#68059](https://github.com/apache/doris/pull/68059) address different parts of this flow; the inspected revisions retain the address mismatch. Validation here was read-only source and public patch inspection. No Doris code was modified, and no build, native test suite, or cloud integration test was run. The reported isolated check counts are author-provided evidence, not independently verified full-system validation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
