Re: How to refresh topics to ingest with KafkaSource?

Hi Mason, I've assigned it to you. Best regards, Martijn On Tue, 2 Nov 2021 at 23:28, Mason Chen wrote: > Hi Arvid, > > I have some bandwidth to contribute to this task and am familiar with the > code. Could you or another committer assign me this ticket? > > Thanks, > Mason > > On Oct 30,

Re: How to refresh topics to ingest with KafkaSource?

Hi Arvid, I have some bandwidth to contribute to this task and am familiar with the code. Could you or another committer assign me this ticket? Thanks, Mason > On Oct 30, 2021, at 5:24 AM, Arvid Heise wrote: > > Hi Mason, > > thanks for creating that. > > We are happy to take contribuitons

Re: How to refresh topics to ingest with KafkaSource?

Hi Mason, thanks for creating that. We are happy to take contribuitons (I flagged it as a starter task). On Wed, Oct 27, 2021 at 2:36 AM Mason Chen wrote: > Hi all, > > I have a similar requirement to Preston. I created > https://issues.apache.org/jira/browse/FLINK-24660 to track this effort.

Re: How to refresh topics to ingest with KafkaSource?

Hi all, I have a similar requirement to Preston. I created https://issues.apache.org/jira/browse/FLINK-24660 to track this effort. Best, Mason > On Oct 18, 2021, at 1:59 AM, Arvid Heise wrote: > > Hi Preston, > > if you still need to set

Re: How to refresh topics to ingest with KafkaSource?

Hi Preston, if you still need to set KafkaSubscriber explicitly, could you please create a feature request for that? For now, you probably have to resort to reflection hacks and build against a the non-public KafkaSubscriber. On Fri, Oct 15, 2021 at 4:03 AM Prasanna kumar <

Re: How to refresh topics to ingest with KafkaSource?

Yes you are right. We tested recently to find that the flink jobs do not pick up the new topics that got created with the same pattern provided to flink kafka consumer. The topics are set only during the start of the jobs. Prasanna. On Fri, 15 Oct 2021, 05:44 Preston Price, wrote: > Okay so

Re: How to refresh topics to ingest with KafkaSource?

Okay so topic discovery is possible with topic patterns, and maybe topic lists. However I don't believe it's possible to change the configured topic list, or topic pattern after the source is created. On Thu, Oct 14, 2021, 3:52 PM Denis Nutiu wrote: > There is a setting for dynamic topic

Re: How to refresh topics to ingest with KafkaSource?

There is a setting for dynamic topic discovery https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/kafka/#topic-and-partition-discovery Best, Denis On Fri, Oct 15, 2021 at 12:48 AM Denis Nutiu wrote: > Hi, > > In my experience with the librdkafka client and the Go

Re: How to refresh topics to ingest with KafkaSource?

Hi, In my experience with the librdkafka client and the Go wrapper, the topic-pattern subscribe is reactive. The Flink Kafka connector might behave similarly. Best, Denis On Fri, Oct 15, 2021 at 12:34 AM Preston Price wrote: > No, the topic-pattern won't work for my case. Topics that I should

Re: How to refresh topics to ingest with KafkaSource?

No, the topic-pattern won't work for my case. Topics that I should subscribe to can be enabled/disabled based on settings I read from another system, so there's no way to craft a single regular expression that would fit the state of all potential topics. Additionally the documentation you linked

Re: How to refresh topics to ingest with KafkaSource?

Hi! I suppose you want to read from different topics every now and then? Does the topic-pattern option [1] in Table API Kafka connector meet your needs? [1] https://ci.apache.org/projects/flink/flink-docs-master/docs/connectors/table/kafka/#topic-pattern Preston Price 于2021年10月14日周四 上午1:34写道：

How to refresh topics to ingest with KafkaSource?

The KafkaSource, and KafkaSourceBuilder appear to prevent users from providing their own KafkaSubscriber. Am I overlooking something? In my case I have an external system that controls which topics we should be ingesting, and it can change over time. I need to add, and remove topics as we refresh