zephyr45 opened a new issue, #13298:
URL: https://github.com/apache/seatunnel/issues/13298

   ### Search before asking
   
   Searched open and closed Kafka transaction/cleanup issues and PRs. #11588 
fixes producer reuse in the cleanup loop; this request concerns duplicate work 
across restoring writers.
   
   ### Description
   
   Track the legacy shared-prefix cleanup optimization deferred from #13060, as 
[requested by 
SEZ9](https://github.com/apache/seatunnel/pull/13060#issuecomment-6936149729).
   
   At `7c272c5a246e617bff1f737fa4e78ed3a1ab7bf5`, 
`KafkaSinkWriter.abortRestoredTransactions` deduplicates transaction prefixes 
only within each writer. Several restoring writers can still probe the same 
legacy `<prefix>-<sequence>` history. 
`KafkaTransactionSender.abortTransaction(prefix, start)` initializes one 
temporary producer per ID and stops at the first unused ID. Initializing that 
ID itself creates broker history, so overlapping scans can extend the range 
visited by later writers. There is no cross-writer cleanup owner or hard probe 
bound; repeated migration retries can repeat the cost.
   
   Reduce redundant broker initialization during legacy checkpoint migration 
while preserving fencing of every unfinished transaction that recovery must 
clean. The current per-writer cleanup must remain until a replacement provides 
equivalent recovery guarantees.
   
   The coordination design needs to cover engine-assigned state (including 
Flink contiguous assignment and Zeta redistribution), mixed historical 
prefixes, cleanup-owner failure and retries. Electing an owner from a 
writer-local state list or current-index modulo alone would not establish that 
every legacy prefix is covered.
   
   ### Usage Scenario
   
   Restore an older checkpoint with a shared transaction prefix at higher 
parallelism, particularly after several failed recovery attempts. Measure 
producer initializations and recovery time while varying writer count, retained 
history and retry count.
   
   Acceptance evidence should include:
   
   - Multi-writer migration tests proving that abandoned legacy records remain 
invisible to `read_committed` consumers and consumption resumes after cleanup.
   - Owner-failure/retry and mixed-prefix cases, with no active sibling 
namespace fenced.
   - A measured reduction in duplicate probes and an explicit complexity/bound 
argument, without assuming a safe fixed scan cap.
   
   ### Related issues
   
   - #13044 / #13060: writer-specific transactional IDs and recovery.
   - #11588: the separate, already-merged cleanup producer-reuse fix.
   
   This issue tracks the deferred optimization; it does not claim that #13060 
implements coordinated cleanup or changes the documented stateless 
`transaction.timeout.ms` boundary. Implementation ownership and the 
coordination design remain open.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to