Re: Issues about removed topics with KafkaSource

2023-11-02 Thread Hector Rios
Hi Emily One workaround that might help is to leverage the state-processor-api[1]. You would have to do some upfront work to create a state-processor job to wipe the state (offsets) of the topic you want to remove and use the newly generated savepoint without the removed state of the topic or topi

Re: Issues about removed topics with KafkaSource

2023-11-01 Thread Emily Li via user
Hey Martijn Thanks for the clarification. Now it makes sense. I saw this feature FLIP-246 is still a WIP and there's no release date yet, and it actually contains quite some changes in it. We noticed there's a WIP PR for this change, just wondering if there's any plan in releasing this feature?

Re: Issues about removed topics with KafkaSource

2023-11-01 Thread Martijn Visser
Hi, That's by design: you can't dynamically add and remove topics from an existing Flink job that is being restarted from a snapshot. The feature you're looking for is being planned as part of FLIP-246 [1] Best regards, Martijn [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageI