Ojasva Jain created KAFKA-21132:
-----------------------------------
Summary: Producer close() hangs indefinitely if the in-flight
transaction abort never succeeds
Key: KAFKA-21132
URL: https://issues.apache.org/jira/browse/KAFKA-21132
Project: Kafka
Issue Type: Bug
Components: clients, producer
Reporter: Ojasva Jain
Assignee: Ojasva Jain
Attachments: AbortCloseBlockingDemoTest.java
h2. Problem
{{KafkaProducer#close()}} (no arguments) is documented to close the producer
and abort any in-flight transaction. If the transaction coordinator is
unreachable, or keeps returning retriable errors while the abort is attempted,
this call never returns and there is no bound on how long it will wait.
h2. Root cause
{{KafkaProducer.close()}} delegates to
{{close(Duration.ofMillis(Long.MAX_VALUE))}}. That method only force-closes the
sender ({{sender.forceClose()}}) if the I/O thread is still alive after waiting
for the given timeout. With an effectively-infinite timeout, that check never
fires, so nothing ever interrupts the shutdown.
On the I/O thread, {{Sender.run()}}'s shutdown path calls
{{transactionManager.beginAbort()}} once, which enqueues a single
{{EndTxnHandler}}, then loops calling {{runOnce()}} to send it. If the
coordinator
is unreachable or returns a retriable error, the handler's {{onComplete()}}
callback just re-enqueues itself ({{reenqueue()}}) with no retry-count or
elapsed-time cap. The loop only exits when the abort actually succeeds or
{{forceClose}} is set and per above, that never happens for no-arg {{close()}}.
So the loop, and {{close()}}, retry forever.
Note: {{close(Duration timeout)}} with a real, finite timeout is unaffected
h2. Repro
Attached {{{}AbortCloseBlockingDemoTest{}}}. It uses a {{MockClient}}, which
answers every {{{}EndTxn{}}}(abort) attempt with {{CONCURRENT_TRANSACTIONS}} (a
retriable error) for many attempts in a row. The test shows
{{producer.close()}} still blocked after several seconds and 15+ retries, only
returning once the mocked coordinator finally accepts the abort.
h2. Expectation
Every transaction already has a bounded lifetime on the broker. After
{{transaction.timeout.ms}}, the coordinator expires it regardless of what the
client does. Given that, the client-side abort attempted during {{close()}}
should never need to wait longer than {{transaction.timeout.ms}}, if the abort
hasn't completed by then, the broker will have already discarded the
transaction on its own, so there is nothing left to wait for. {{close()}}
should not block past that point.
h2. Impact
Any caller of {{close()}} without args (a very common pattern) can hang forever
if the transaction coordinator is unreachable or persistently erroring at
shutdown time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
