jaredmdobson opened a new pull request, #70726:
URL: https://github.com/apache/doris/pull/70726
### Background
In cloud mode the FE keeps no transaction state of its own:
`CloudGlobalTransactionMgr` asks the meta service. `getTransactionId(dbId,
label)` resolves a label to its latest transaction and backs `SHOW TRANSACTION
WHERE LABEL = '...'` (which reports `null` as "transaction with label ... does
not exist") and the broker-load label check. `getTransactionIdByLabel(dbId,
label, statuses)` backs stream load 2PC commit and abort by label. The meta
service answers `get_txn` and `get_txn_id` with `TXN_ID_NOT_FOUND` when the
label has no transaction, and with other codes (`KV_TXN_GET_ERR`,
`MAX_QPS_LIMIT`, ...) or an RPC failure when the lookup itself failed.
### What problem does this PR solve?
Issue Number: #70586
Related PR: #41718 (introduced `getTransactionId` for cloud SHOW TRANSACTION)
PR Type: Bugfix
Problem Summary:
In cloud mode, `SHOW TRANSACTION FROM db WHERE LABEL = 'l'` can answer
`transaction with label l does not exist` for a label whose transaction exists
and is VISIBLE: `getTransactionId` catches every lookup failure and returns
`null`. We saw this on 4.2.0-rc03 for labels a few minutes old, while `GET
/api/<db>/get_load_state?label=l` returned VISIBLE for the same labels. A
loader that trusts the answer resubmits the label and gets
`LABEL_ALREADY_EXISTS`. Stream load 2PC by label has the same flaw. After this
PR only the meta service's `TXN_ID_NOT_FOUND` means "not found"; every other
failure is reported as an error with the meta service's message.
### Details
- **Frequency:** Intermittent. Triggers whenever the meta-service lookup
fails or is throttled (RPC failure after the proxy's retries, `KV_TXN_GET_ERR`,
`MAX_QPS_LIMIT`, `KV_TXN_CONFLICT` after three tries, ...), and, for a label
still in flight, when the label's status changes between the two calls
`getTransactionId` made. We hit it twice in one evening on a stream load
workload that checks labels often; the FE log had rotated, so which failure it
was is inferred from the code, not observed.
- **Root cause:** `CloudGlobalTransactionMgr.getTransactionId` called
`getLabelState` and then `getTransactionIdByLabel` inside `try { ... } catch
(UserException e) { return null; }`, so a failed lookup and an absent label
both came back as `null`. `getTransactionIdByLabel` turned an `RpcException`
and every non-OK `get_txn_id` code into `TransactionNotFoundException`.
- **Why existing tests missed it:** `CloudGlobalTransactionMgrTest` has no
case for `getTransactionId`, `getLabelState` or `getTransactionIdByLabel`. Its
other cases mock `MetaServiceProxy`, but only on the commit, begin and abort
paths. `query_p0/show/test_show_transaction` checks only the success path
against a healthy meta service, and no regression suite can make the meta
service fail a lookup on demand.
- **Fix:** `getTransactionId` reads the label with one `get_txn` call (the
same call `getLabelState` makes; its `TxnInfoPB` already carries the latest
transaction id), returns `null` only for `TXN_ID_NOT_FOUND`, and throws
`AnalysisException` with the meta service's message otherwise. The interface
method already declares `throws AnalysisException`, and `SHOW TRANSACTION` and
`BrokerLoadJob` already handle it. `getTransactionIdByLabel` keeps
`TransactionNotFoundException` for `TXN_ID_NOT_FOUND` and throws
`UserException` for other failures. In the meta service, `get_txn` by label now
answers `TXN_ID_NOT_FOUND` for a label that holds only its version stamp:
`begin_txn` commits the label that way before it writes the label's first
transaction, and `get_txn` answered `PROTOBUF_PARSE_ERR` in that window, where
`get_txn_id` already answers `TXN_ID_NOT_FOUND`. Without it, the FE change
would turn that normal window into an error. `BrokerLoadJob.findSelfTxnByLabel`
still catches and logs
any lookup error, so broker load behaves as before.
- **Behavior change:** a failed lookup is now an error with the meta
service's message, where it was `transaction with label ... does not exist`
(SHOW TRANSACTION) or `transaction not found, label=...` (2PC by label). A
missing label answers as before.
- **Other paths and branches:** `getTransactionState(dbId, txnId)` has the
same pattern: it returns `null` on any lookup failure, and commit reports that
as a missing transaction (#70586). SHOW TRANSACTION and 2PC call it after
resolving the label, so a failure there can still read as "does not exist". It
is left for a follow-up because its `null` contract is shared with
shared-nothing mode and many callers. Shared-nothing `GlobalTransactionMgr`
reads local memory and is not affected. `getTransactionId` has swallowed errors
since #41718 (3.0.3). Checked: branch-3.1, branch-4.0, branch-4.1 and
branch-4.2 all have the bug.
### Testing
New FE unit tests in `CloudGlobalTransactionMgrTest`, with
`MetaServiceProxy` mocked:
- `testGetTransactionIdByLabelFailsOnLookupError` (`KV_TXN_GET_ERR`,
`MAX_QPS_LIMIT`, `INVALID_ARGUMENT`, `UNDEFINED_ERR`, `RpcException`): expects
`AnalysisException`. Before the fix `getTransactionId` returned `null`.
- `testGetTransactionIdByMissingLabelReturnsNull`: `TXN_ID_NOT_FOUND` still
gives `null`.
- `testGetTransactionIdReadsTxnIdFromGetTxn` (PREPARED, PRECOMMITTED,
COMMITTED, VISIBLE, ABORTED): the id comes from one `get_txn` call by label,
and `get_txn_id` is never called.
- `testGetTransactionIdByLabelWithStatusReportsOnlyMissingAsNotFound`: only
`TXN_ID_NOT_FOUND` gives `TransactionNotFoundException`.
New meta-service unit test `MetaServiceTest.GetTxnByLabelTest`: from the
`begin_txn:after:commit_txn:1` sync point (the label committed with only its
version stamp), `get_txn` and `get_txn_id` by label both answer
`TXN_ID_NOT_FOUND`; before the fix `get_txn` answered `PROTOBUF_PARSE_ERR`.
After `begin_txn`, `get_txn` by label returns the new transaction.
None of these were run locally (no local FE or cloud build); this PR relies
on the FE UT and cloud UT pipelines.
### Release note
In cloud mode, `SHOW TRANSACTION ... WHERE LABEL = '...'` and stream load
2PC by label now report a failed meta-service lookup as an error, instead of
saying the transaction does not exist.
### Check List (For Author)
- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Partially verified: the FE and meta-service unit
tests have not been run locally (no local build); CI runs them.
### AI Disclosure
- Tool: Claude Code (Claude Opus 5.5)
- Generated: code, tests, this description
- Human review: pending; the author reviews the generated code and tests
before this PR leaves draft
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]