lnbest0707 commented on PR #279: URL: https://github.com/apache/flink-connector-kafka/pull/279#issuecomment-5516369094
> > Hi. Sorry for delayed response. I was on vacation. I left 1 question-comment. If my understanding is correct, we may introduce duplicate producing which will affect EOS. > > p.s. During my vacation I also missed another PR #278 . That's why I was curious and done a bit of deep dive into the Dynamic topic. It looks like we may have a dataloss for DynamicKafka workflow. Worth to check my [PR](https://github.com/apache/flink-connector-kafka/pull/295) (I know you contribute a lot to DynamicKafka, maybe you can review?) because it also fixes that EOS issue. It is not a duplicate PR, it solves a different issue. > > The #295 looks valid. And your question also makes sense. The approach in the PR has been tuned a few times due to various corner cases. Let me also rethink about whether we can get some better approaches to make it simpler and catch all issues. The updated method is by holding the new cluster consumption till the next checkpoint to avoid all those corner cases. I also tried another approach by forcing a full rebalance/restart upon the events. The alternative method also works but is more complicated and more interruptive on the jobs. So IMO, the current one might be the path to go. @Savonitar WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
