SEZ9 commented on PR #13060:
URL: https://github.com/apache/seatunnel/pull/13060#issuecomment-6593944491
@zephyr45 Thanks for the ledger and for reproducing the extra Flink
counterexample before touching production code — that is exactly the order I
was hoping for.
On **F1**: the move from checkpoint-qualified relocation to a
checkpoint-independent recovery revision (`<base>_r<revision>w<current-index>`)
sounds like the right shape. Same-recovery retries deriving the same namespace,
while a later recovery advances the checkpointed revision, addresses the
writer-local counter restart that the `3 -> 4 -> 3 -> 4 -> 3` regression
exposed (the duplicate `shared-prefix-7-gtestw3r2c1` case). Good that
`FlinkSinkWriterTest` records that fresh and restored writers may legitimately
diverge in their counters. Two follow-ups so I can close this one out:
- Please confirm the new regression fails on the previous logic and passes
on `97516b977` in the same test run (i.e. it is a true red/green, not only
green).
- Please point me at where the revision is read from and written back to
`KafkaSinkState`, so I can verify the "later recovery sees and advances the
revision" step without guessing.
On **F2**: parsing each `KafkaSinkState` relative to its own saved
`transactionIdPrefix` is the fix I wanted. Your comment appears to have been
cut off mid-sentence here, though ("no common-prefix assumpti…"), so I only
have the first half of that item.
Could you repost the rest of the ledger? Specifically I am missing the
remainder of **F2**, and the entries for **F3** (scale-up writer starting its
sequence at 1), **F4** (Spark taskId as subtask index), **F5/F6** (concurrent
legacy shared-namespace fencing), and **F7** (aligning the `transaction_prefix`
option row and the engine scope of the rescaling claim in
`docs/en/connectors/sink/Kafka.md`). Once I have those, I will do the final
pass against `97516b977`.
<!-- streview-comment:2239 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
