Bill Bejeck created KAFKA-21466:
-----------------------------------
Summary: Handle producer commit timeouts under EOS without wiping
state stores (KIP-892)
Key: KAFKA-21466
URL: https://issues.apache.org/jira/browse/KAFKA-21466
Project: Kafka
Issue Type: Improvement
Components: streams
Reporter: Bill Bejeck
Assignee: Bill Bejeck
Fix For: 4.5.0
Under EOS, if {{producer.commitTransaction()}} times out ( \{{max.block.ms}}
exceeded), Kafka Streams treats it as task corruption. It throws
{{{}TaskCorruptedException{}}}, wipes the affected tasks' local state stores,
and restores them from the changelog. For large stores that restore can be very
expensive.
KIP-892 transactional state stores keep uncommitted writes in a transaction
buffer, and the buffer is only flushed to the underlying store on a successful
commit. So a commit timeout no longer needs a wipe. Streams can drop the
uncommitted transaction buffer, abort the transaction, and resume from the last
committed offsets, with the local store still consistent with the committed
changelog position.
However, we should not create special handling of the
{{TaskCorruptedException. }} Instead, we'll add a new exception that tells the
runtime to discard the uncommitted transactional state and retry without wiping
the store. A new exception type is a public API change, so this needs a KIP.
Scope:
- Add a new exception type for "drop the uncommitted transaction, no wipe"
recovery.
- On a producer commit timeout under EOS with transactional stores, discard
the transaction buffers, abort the transaction, and reprocess from the last
committed offsets.
- Keep the current wipe-and-restore behavior for stores that aren't
transactional, and wherever the store's consistency can't be guaranteed.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)