MartijnVisser commented on PR #314: URL: https://github.com/apache/flink-connector-kafka/pull/314#issuecomment-5623039983
Concept is right and I would like this to land. Three things from looking at the surrounding design. Measured against a broker: with an unchanged prefix the orphan is cleared at the first checkpoint after the fenced commit, because POOLING takes the id back and `initTransactions` aborts it. It only survives when the id is never recycled: a changed `transactionalIdPrefix`, as `rescaleListing` does, or a downscale, where new transactions only use `ownedSubtaskIds[0]` so an orphan under a secondary owned subtask id is never reused. The downscale case needs no user error and is the strongest justification for the change. Worth putting in the description and in the FLINK-40626 release note in place of the "~1 hour" framing. On the live owner question above: the window is narrow, but it ends in silent data loss rather than a failover, since a fenced commit is `signalFailedWithKnownReason` and the committable is discarded. One `describeTransactions` before the bump closes it: abort only when the open transaction started before this committer recovered. FLINK-39218 (#296) adds the same epoch bump as a CLI abort, with a plain `KafkaProducer` and `max.block.ms` at 10s. One shared helper in the connector beats two implementations, and it settles the `max.block.ms` comment above. @Savonitar does that work for you? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
