Ritika Reddy created KAFKA-21181:
------------------------------------
Summary: Transaction coordinator could return INVALID_TXN_STATE
from stale cache after losing __transaction_state leadership
Key: KAFKA-21181
URL: https://issues.apache.org/jira/browse/KAFKA-21181
Project: Kafka
Issue Type: Bug
Affects Versions: 4.3.0, 4.2.0
Reporter: Ritika Reddy
Assignee: Ritika Reddy
Fix For: 4.3.0, 4.2.0
When a broker loses leadership of a __transaction_state partition, its
TransactionStateManager keeps the partition's cache entry until
TransactionCoordinator.onResignation runs. In
BrokerMetadataPublisher.onMetadataUpdate, that happens after
replicaManager.applyDelta and after the group coordinator update. So there's a
window in which the broker is already a follower for the partition but still
answers coordinator requests from its old cache.
TransactionStateManager.getAndMaybeAddTransactionState (trunk lines 400-421)
only checks loadingPartitions and transactionMetadataCache; it never checks
that the broker still leads the partition. Answers that require a log write are
safe, because the acks=-1 append fails with NOT_LEADER_OR_FOLLOWER, which is
mapped to NOT_COORDINATOR. Answers decided purely from the cache are not.
With TV2 this surfaces as a fatal error for the producer:
1. Transaction N commits on coordinator A. The cache on A holds COMPLETE_COMMIT
at epoch E, and the client now has epoch E.
2. Leadership of the __transaction_state partition moves to broker B. B loads
the partition, and the partition leaders' implicit AddPartitionsToTxn for
transaction N+1 go to B, which moves the transaction to ONGOING.
3. The client hasn't had any reason to rediscover the coordinator: under TV2
its first coordinator request in the transaction is EndTxn. So it sends
EndTxn(COMMIT, E) to A.
4. A is already a follower but hasn't resigned yet. It answers from the cache:
COMPLETE_COMMIT + COMMIT at the current epoch is not a retry, so it returns
INVALID_TXN_STATE without writing to the log:
TransactionalId: <id>'s state is COMPLETE_COMMIT, but received transaction
marker result to send: COMMIT
5. The producer treats INVALID_TXN_STATE as fatal. The transaction on B stays
ONGOING until it times out.
The correct response is NOT_COORDINATOR, which would make the client rediscover
B and commit. We've seen gaps of several seconds between the broker becoming
follower and resigning, and in one case ~37 s, with no broker pause involved.
Other cache-only answers are exposed the same way, for example PRODUCER_FENCED
/ INVALID_PRODUCER_ID_MAPPING, the verify-only AddPartitionsToTxn path, and the
"partitions already added" shortcut.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
