[
https://issues.apache.org/jira/browse/KAFKA-21181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ritika Reddy updated KAFKA-21181:
---------------------------------
Affects Version/s: 4.1.0
4.0.0
> Transaction coordinator could return INVALID_TXN_STATE from stale cache after
> losing __transaction_state leadership
> -------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-21181
> URL: https://issues.apache.org/jira/browse/KAFKA-21181
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 4.0.0, 4.1.0, 4.2.0, 4.3.0
> Reporter: Ritika Reddy
> Assignee: Ritika Reddy
> Priority: Major
> Fix For: 4.0.0, 4.1.0, 4.2.0, 4.3.0
>
>
> When a broker loses leadership of a __transaction_state partition, there is a
> *race* between the replica state change and the transaction coordinator's
> resignation. BrokerMetadataPublisher.onMetadataUpdate applies the delta to
> ReplicaManager first, which makes the broker a follower. It then updates the
> group coordinator, and only after that calls
> TransactionCoordinator.onResignation, which drops the partition's entry from
> the TransactionStateManager cache. Until resignation runs, the broker is a
> follower for the partition but still answers coordinator requests from its
> old cache. The window is usually short, but its length depends on how much
> work the publish does before it reaches the transaction coordinator, and it
> can stretch to seconds.
> TransactionStateManager.getAndMaybeAddTransactionState only checks
> loadingPartitions and transactionMetadataCache; it never checks that the
> broker still leads the partition. Answers that require a log write are safe:
> the acks=-1 append fails with NOT_LEADER_OR_FOLLOWER, which is mapped to
> NOT_COORDINATOR. Answers decided purely from the cache are not, so any
> request that hits the old coordinator inside the window can get a wrong
> answer.
> {*}With TV2{*}, one way to lose this race is a fatal producer error:
> 1. Transaction N commits on coordinator A. A's cache holds COMPLETE_COMMIT at
> epoch E, and the client now has epoch E.
> 2. Leadership of the __transaction_state partition moves to broker B. B loads
> the partition, and the partition leaders' implicit AddPartitionsToTxn for
> transaction N+1 go to B, which moves the transaction to ONGOING.
> 3. The client has had no reason to rediscover the coordinator: under TV2,
> EndTxn is its first coordinator request in the transaction. So it sends
> EndTxn(COMMIT, E) to A.
> 4. If this request reaches A after A became a follower but before it
> resigned, A answers from its cache. COMPLETE_COMMIT + COMMIT at the current
> epoch is not a retry, so A returns INVALID_TXN_STATE without writing to the
> log:
> TransactionalId: <id>'s state is COMPLETE_COMMIT, but received transaction
> marker result to send: COMMIT
> 5. The producer treats INVALID_TXN_STATE as fatal. The transaction on B stays
> ONGOING until it times out (or the producer re-initializes).
> If the EndTxn arrives after A resigns, A returns NOT_COORDINATOR, and the
> client rediscovers B and commits normally. The outcome depends only on which
> side of that window the request lands. It doesn't need a broker pause or any
> fault: an ordinary leadership move (reassignment, controlled shutdown,
> preferred leader election) is enough, and producers that commit often are the
> most exposed.
> Other answers decided purely from the cache race the same way, for example
> PRODUCER_FENCED / INVALID_PRODUCER_ID_MAPPING, the verify-only
> AddPartitionsToTxn path, and the "partitions already added" shortcut.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
