[
https://issues.apache.org/jira/browse/KAFKA-20716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18097780#comment-18097780
]
Qiu Yanjun commented on KAFKA-20716:
------------------------------------
Hi, I traced the stuck LSO through the current code. After the unclean
election, the partition rebuilds the transactional data as an ongoing
transaction from its local log, but the missing COMMIT/ABORT marker means
`ProducerStateManager.firstUnstableOffset` remains at that transaction
indefinitely; ongoing transactions do not expire. The existing
`kafka-transactions.sh find-hanging` logic treats a locally open transaction as
hanging when the coordinator description no longer contains the partition, and
the partition-local `abort --start-offset` path can write an ABORT marker and
unblock the LSO.
Before proposing an automatic broker change, could maintainers clarify the
intended acceptance criteria? Should this Jira be resolved by the existing
operator recovery (plus documentation), or should Kafka automatically reconcile
after an unclean election? If automatic reconciliation is desired, when the
coordinator has already reached CompleteCommit/CompleteAbort but no longer
exposes the target partitions, should the missing local marker preserve the
coordinator result (COMMIT/ABORT) or may recovery conservatively ABORT, and
which component should retain or rediscover those partitions?
I do not think changing LSO to ignore the old transaction is safe because that
would expose undecided data to READ_COMMITTED. I am holding off on a patch
until the recovery semantics are explicit.
> LSO stuck after unclean leader election
> ---------------------------------------
>
> Key: KAFKA-20716
> URL: https://issues.apache.org/jira/browse/KAFKA-20716
> Project: Kafka
> Issue Type: Bug
> Reporter: Luke Chen
> Priority: Major
>
> When a topic has an unclean leader election, the new leader might contain txn
> data without COMMIT/ABORT markers. However, the data in __transaction_state
> shows the transaction is committed/aborted, so the transaction timeout will
> not expire here. This causes the LSO stuck and READ_COMMITTED will never
> proceed.
>
> reproduce steps:
> 1. Create a cluster with 2 brokers
> 2. Create a topic with unclean leader election enabled
> {code:java}
> bin/kafka-topics.sh --create --topic t1 --bootstrap-server localhost:9091
> --replication-factor 2 --config unclean.leader.election.enable=true {code}
> 3. write a txn record to the topic t1, but don't commit it, yet.
> 4. Before the record committed in step (3), shutdown the follower broker
> (suppose it's broker 2)
> 5. commit the step (3) record.
> 6. Now, the the topic t1-0 in broker 1 contains [offset 0 (data) and offset 1
> (commit)], but broker 2 only contains [offset 0 (data)]
> 7. shutdown broker 1, so both broker 1 and 2 are down, but broker 2 is not
> the last leader or ELR
> 8. start up broker 2, unclean leader election triggered
> 9. start up broker 1, log truncation on t1-0, so the log becomes [offset 0
> (data)]
> 10. appending more non-txn data to t1-0
> 11. consume with READ_COMMITTED, it'll return nothing.
>
> We never document anywhere about unclean leader election unsupported in
> transaction feature, I think this should be supported and we have to find out
> a solution for it.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
