Doris-Breakwater commented on issue #70586: URL: https://github.com/apache/doris/issues/70586#issuecomment-6884967587
Breakwater-GitHub-Analysis-Slot: slot_8a79ab2cff59 **Initial assessment:** The error-classification problem is supported by source inspection in both public `4.1.5-release` (`917b20be83878e98f94e3ebf7efccc80346e3e50`) and the cited master commit. A failed lookup can become a false transaction-not-found response. This verifies the code mechanism; it does not establish the deployed transaction's final state, expiry, or the cause of the KV failure. The proposed fault injection has not been executed in this investigation. Suggested maintainer routing: cloud FE transaction/error handling. The issue currently has no labels. **Verified code path** - In 4.1.5, `getTransactionState` returns `null` for `RpcException`, a null response, any non-OK status, or missing `txn_info`. These outcomes therefore lose their distinction from absence. See [release lookup handling](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/fe/fe-core/src/main/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgr.java#L2391-L2428). - More precisely, the exception in the issue's master lines 483–486 is inside `checkTransactionStateBeforeCommit`, reached through the commit path. The release has the same null-to-`TransactionCommitFailedException` conversion. On the ordinary commit path, this check is gated by a nonempty list of participating unique-key Merge-on-Write tables, derived from matching tablet commit information. A disposable table alone is therefore insufficient to guarantee this reproduction. See [release caller and check](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/fe/fe-core/src/main/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgr.java#L430-L505), [table selection](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/fe/fe-core/src/main/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgr.java#L962-L986), and [pinned master check](https://github.com/apache/doris/blob/42b148bac33f798edc9f8ae984a462b1b4a4f472 /fe/fe-core/src/main/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgr.java#L478-L486). - Meta Service already distinguishes absence: reading a missing transaction-info key yields `TXN_ID_NOT_FOUND`; other read errors use the KV error mapping. The FE discards this distinction. See [release `get_txn`](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/cloud/src/meta-service/meta_service_txn.cpp#L3790-L3812). - The server retry wrapper can translate exhausted `KV_TXN_STORE_GET_RETRYABLE` into `KV_TXN_GET_ERR`. The FE RPC wrapper returns that status without an additional status-based retry for it. Thus the approximately 1 ms between the FE error logs does not establish total lookup duration or whether server retries occurred. `RetryableNotCommitted` describes a KV operation, not the Doris transaction's settled state. See [server retry handling](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/cloud/src/meta-service/meta_service.h#L1072-L1132) and [FE RPC handling](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/fe/fe-core/src/main/java/org/apache/doris/cloud/rpc/MetaServiceProxy.java#L244-L310). **Information needed to verify the incident** 1. Exact FE, BE, and Meta Service build hashes, including any patches or version skew; the version string alone cannot prove equivalence to the public tag. 2. Table DDL, ingestion/commit API, transaction mode, minimal commands, and client retry/rollback behavior. For the identified path, confirm a participating unique-key MoW table and nonempty matching tablet commit information. 3. Sanitized, correlated FE and Meta Service logs covering the lookup and all commit/abort attempts for the same database, transaction, and label, with timestamps/time zones, complete status messages, and exception stacks. Include corresponding BE/client responses and raw underlying KV/FDB error codes. Preserve correlation while redacting credentials and sensitive identifiers. 4. Actual server retry settings (`enable_txn_store_retry`, `txn_store_retry_times`, `txn_store_retry_base_intervals_ms`), RPC/client deadlines, and the transaction/label state after recovery. These are diagnostic inputs; increasing retries or transaction timeouts is not yet justified. **Recommended next steps** - Run the proposed fault injection on an isolated cloud test cluster with a unique-key MoW table and actual written rows. Intercept the specific pre-commit lookup and return `KV_TXN_GET_ERR` while retaining the transaction record. Record whether this invocation sends a subsequent `commit_txn`; this failure branch should stop before that call. Earlier or concurrent attempts still require settlement checks. Compare with `TXN_ID_NOT_FOUND`, then remove the fault and inspect the same transaction/label. - Preserve a distinct lookup-failed outcome, including the original Meta Service status/message or transport cause. Reserve not-found for explicit absence; treat missing response/transaction information as lookup/protocol failure. Review callers before changing the shared nullable API; status-query and other commit entry points also need to preserve the distinction through the client boundary. - Add regression coverage for lookup failure versus definite absence, RPC failure, missing response information, and recovery to PREPARE/COMMITTED/VISIBLE/ABORTED. Preserve the existing handling of already committed/visible transactions. Any retry/settlement should retain the same transaction and label, respect the existing deadline, and avoid inferring that replay under a new identity is safe from this error alone. No code changes or runtime reproduction were performed. The remaining uncertainty is incident correlation and the underlying KV failure, not the demonstrated loss of error information in the inspected FE code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
