DanielLeens commented on issue #13298:
URL: https://github.com/apache/seatunnel/issues/13298#issuecomment-6949673631

   Classification: F / Connector-V2 Kafka legacy transactional-recovery cleanup.
   
   Thanks for restoring this follow-up and for keeping it separate from #13060. 
I rechecked remote `dev` at `50cf6b6711f9316557daa3a5e17defdab4247953` and the 
current #13060 head `7c272c5a246e617bff1f737fa4e78ed3a1ab7bf5`. On `dev`, 
`KafkaSinkWriter` starts recovery cleanup from its locally assigned 
`KafkaSinkState` list and asks `KafkaTransactionSender` to probe 
`<prefix>-<sequence>` IDs until it initializes the first unused ID. #13060 is 
still open; its current design deliberately keeps that legacy shared-prefix 
cleanup per writer and explicitly defers F5/F6 cross-writer coordination. So 
this tracker is valid, but it must remain a follow-up rather than being folded 
into or implemented in parallel with #13060.
   
   Please add a short design note here before opening a code PR, after #13060 
has merged or has otherwise established its final recovery contract. In 
particular, it needs to define:
   
   1. the persisted, engine-neutral input that determines cleanup coverage and 
the per-prefix owner; a writer-local list or current-index/modulo rule cannot 
prove all legacy prefixes are covered;
   2. owner failure, retry, and restore behavior, including how a replacement 
can safely resume cleanup without leaving a sibling namespace fenced or an 
unfinished legacy transaction visible;
   3. mixed historical prefixes and the actual Flink and Zeta redistribution 
cases, rather than assuming one assignment scheme;
   4. a termination and complexity argument that reduces redundant producer 
initialization without an arbitrary scan cap that could skip an unfinished 
transaction; and
   5. an executable proof matrix: at least two restoring writers, shared and 
mixed prefixes, owner failure/retry, `read_committed` visibility for abandoned 
versus new records, and recorded producer-initialization/recovery-time 
measurements.
   
   Keep the existing per-writer cleanup until that replacement proves 
equivalent fencing coverage. The documented no-retained-state 
`transaction.timeout.ms` boundary is separate and should not be broadened by 
this optimization. Once the design has maintainer agreement, a focused 
Connector-V2 PR with the corresponding recovery tests will be the right next 
step.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to