zephyr45 opened a new issue, #13298: URL: https://github.com/apache/seatunnel/issues/13298
### Search before asking Searched open and closed Kafka transaction/cleanup issues and PRs. #11588 fixes producer reuse in the cleanup loop; this request concerns duplicate work across restoring writers. ### Description Track the legacy shared-prefix cleanup optimization deferred from #13060, as [requested by SEZ9](https://github.com/apache/seatunnel/pull/13060#issuecomment-6936149729). At `7c272c5a246e617bff1f737fa4e78ed3a1ab7bf5`, `KafkaSinkWriter.abortRestoredTransactions` deduplicates transaction prefixes only within each writer. Several restoring writers can still probe the same legacy `<prefix>-<sequence>` history. `KafkaTransactionSender.abortTransaction(prefix, start)` initializes one temporary producer per ID and stops at the first unused ID. Initializing that ID itself creates broker history, so overlapping scans can extend the range visited by later writers. There is no cross-writer cleanup owner or hard probe bound; repeated migration retries can repeat the cost. Reduce redundant broker initialization during legacy checkpoint migration while preserving fencing of every unfinished transaction that recovery must clean. The current per-writer cleanup must remain until a replacement provides equivalent recovery guarantees. The coordination design needs to cover engine-assigned state (including Flink contiguous assignment and Zeta redistribution), mixed historical prefixes, cleanup-owner failure and retries. Electing an owner from a writer-local state list or current-index modulo alone would not establish that every legacy prefix is covered. ### Usage Scenario Restore an older checkpoint with a shared transaction prefix at higher parallelism, particularly after several failed recovery attempts. Measure producer initializations and recovery time while varying writer count, retained history and retry count. Acceptance evidence should include: - Multi-writer migration tests proving that abandoned legacy records remain invisible to `read_committed` consumers and consumption resumes after cleanup. - Owner-failure/retry and mixed-prefix cases, with no active sibling namespace fenced. - A measured reduction in duplicate probes and an explicit complexity/bound argument, without assuming a safe fixed scan cap. ### Related issues - #13044 / #13060: writer-specific transactional IDs and recovery. - #11588: the separate, already-merged cleanup producer-reuse fix. This issue tracks the deferred optimization; it does not claim that #13060 implements coordinated cleanup or changes the documented stateless `transaction.timeout.ms` boundary. Implementation ownership and the coordination design remain open. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
