Bill Bejeck created KAFKA-21466:
-----------------------------------

             Summary: Handle producer commit timeouts under EOS without wiping 
state stores (KIP-892)
                 Key: KAFKA-21466
                 URL: https://issues.apache.org/jira/browse/KAFKA-21466
             Project: Kafka
          Issue Type: Improvement
          Components: streams
            Reporter: Bill Bejeck
            Assignee: Bill Bejeck
             Fix For: 4.5.0


Under EOS, if {{producer.commitTransaction()}} times out ( \{{max.block.ms}} 
exceeded), Kafka Streams treats it as task corruption. It throws 
{{{}TaskCorruptedException{}}}, wipes the affected tasks' local state  stores, 
and restores them from the changelog. For large stores that restore can be very 
expensive.  

                                                                 
  KIP-892 transactional state stores keep uncommitted writes in a transaction 
buffer, and the buffer is only flushed to the underlying store on a successful 
commit. So a commit timeout no longer needs a wipe. Streams can drop the 
uncommitted transaction buffer, abort the transaction, and resume from the last 
committed offsets, with the local store still consistent with the committed 
changelog position.                                                             
                                                                                
                                                                                
       However, we should not create special handling of the 
{{TaskCorruptedException. }} Instead, we'll add a new exception that tells the 
runtime to discard the uncommitted transactional state and retry without wiping 
the store.  A new exception type is a public API change, so this needs a KIP.   
       
                                                                                
                                                                              
Scope:
  - Add a new exception type for "drop the uncommitted transaction, no wipe" 
recovery.
  - On a producer commit timeout under EOS with transactional stores, discard 
the transaction buffers, abort the transaction, and reprocess from the last 
committed offsets.
  - Keep the current wipe-and-restore behavior for stores that aren't 
transactional, and wherever the store's consistency can't be guaranteed.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to