tuantran0910 opened a new issue, #68755: URL: https://github.com/apache/doris/issues/68755
### Search before asking - [x] I searched existing issues and PRs and found no duplicate for this coordinator-ID/FQDN mismatch. Related reports and fixes are listed below. ### Version Observed on our Doris 4.1.3-based fork. The local fix is commit [b472de869855d9a3a3db3ac62c5b17bbea1d18a8](https://github.com/thealtoclef/doris/commit/b472de869855d9a3a3db3ac62c5b17bbea1d18a8), based on parent `6be60d6aa440e5fb1638abb17cd6e9e67047c5f3`. The affected address-matching logic is also present in upstream master `7ac0eb2fdb426595f810e94ab77b7aa2eece12d5` and branch-4.1 `78aa9c996accd114a6e0f5943d7882ee1a34011a`, inspected during investigation. ### What's Wrong? In cloud mode with FE FQDN addressing (`enable_fqdn_mode`), a stream-load transaction can remain in PREPARE after its coordinator BE is restarted/rescheduled. Schema-change and rollup jobs then remain in WAITING_TXN because the abandoned transaction is not recognized as belonging to the failed coordinator. Our local fix commit records an affected cluster where the jobs had to be unblocked by manually aborting leaked transactions. The default stream-load timeout is three days, so normal expiration can leave DDL blocked for hours or days. There are two related identity checks: 1. FE records a stream-load coordinator using the backend ID, the connection's client IP, and backend startup time. `GlobalTransactionMgr.checkFailedTxnsByCoordinator()` looks up the backend by ID but additionally requires `be.getHost().equals(coordinator.ip)` before checking restart/liveness. An FQDN does not equal the recorded pod IP. Finding the backend sets `offline=false`, so the address mismatch causes the method to return false despite a restart. 2. Meta-service `get_prepare_txn_by_coordinator()` requires IP equality, and `abort_txn_with_coordinator()` requires both ID and IP equality. Heartbeat cleanup passes the BE host. A hostname/IP mismatch, or a changed pod IP after rescheduling, prevents matching old prepared transactions even when their backend ID is unchanged. For example, the old transaction records `(backend ID=123, IP=A, startup=T1)`. After rescheduling, the retained backend registration has `(ID=123, hostname=be-0.example, startup=T2)`, with `T2>T1` and a new pod IP B. The existing address checks miss the old transaction. ### What You Expected? For a retained BE registration, coordinator identity should be determined by its backend ID. A restarted or sufficiently long unavailable coordinator's abandoned transactions should be aborted automatically, allowing waiting schema-change and rollup jobs to progress. Healthy transactions belonging to the new BE process must remain protected by startup-time filtering. Committed/final transactions must not be selected for abort, and concurrent cleanup must not cancel DDL jobs. ### How to Reproduce? Suggested controlled integration scenario (the new end-to-end scenario has not yet been run): 1. Run a cloud deployment with FQDN addressing and both `enable_abort_txn_by_checking_coordinator_be` and `enable_abort_txn_by_checking_conflict_txn` enabled. 2. Create a table and start a stream load. Hold it before commit so its transaction remains PREPARE; the existing `StreamLoadExecutor.commit_txn.block` debug point in `test_abort_txn_by_be.groovy` can help create this state. 3. Start a heavyweight schema change or rollup on that table and verify WAITING_TXN. 4. Restart/reschedule the coordinating BE while preserving its FQDN, heartbeat port, and FE backend registration/ID. A changed pod IP demonstrates the unstable-address case. 5. Verify the BE startup time advances. Without the identity fix, the old prepared transaction is not selected by the address-based cleanup checks, and the DDL remains waiting until transaction expiration or manual abort. A smaller deterministic meta-service reproduction is to create prepared BE transactions with ID 123 and address A, then call `get_prepare_txn_by_coordinator` with ID 123 and address B. Current upstream returns no matches; ID-based matching returns the old prepared transactions. ### Anything Else? Proposed fix: when both coordinator IDs are nonzero, match by backend ID; fall back to the address when an ID is unknown. In the cloud FE failed-coordinator check, remove the redundant host/IP guard after backend lookup by ID, retaining restart/liveness checks. Preserve non-cloud FE behavior. The local patch adds a same-ID/different-IP meta-service regression case. Isolated compiled checks passed for 54 identity combinations and 33 FE failure-detection combinations. Full FE validation is running in Docker; no successful full-suite or new end-to-end result is claimed here. Related upstream work: - #29708 reports the same WAITING_TXN/three-day symptom but a rollback-lock-timeout cause. - #37669 introduced heartbeat and schema-change conflict cleanup. - #65196 and #67661 protect schema-change/rollup recovery from abort failures; #67661 also excludes committed/final transactions. These complement the identity fix. - #67761 / #68059 improve coordinator-query scan reliability and handle KV_TXN_TOO_OLD, but retain IP equality and do not fix this identity mismatch. ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! (A candidate fix is available in the fork linked above.) ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
