Re: Flink handle both kafka source and db source

2021-10-30 Thread Arvid Heise
If you are using JDBCInputFormat, then the respective query is executed only once. The task will be closed and with it any downstream tasks that do not have multiple inputs. However, JDBCInputFormat currently cannot be used with HybridSource. Only source that use the new unified Source interface c

Re: Flink handle both kafka source and db source

2021-10-27 Thread Qihua Yang
Thank you for your reply! I will check Hybrid source. How do we know the database table is fully scanned? And after the scan is completed, does flink scan the table again or keep idling? Thanks, Qihua On Tue, Oct 26, 2021 at 1:59 PM Rafi Aroch wrote: > Hi, > > Take a look at the new 1.14 featur

Re: Flink handle both kafka source and db source

2021-10-26 Thread Rafi Aroch
Hi, Take a look at the new 1.14 feature called Hybrid Source: https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/connectors/datastream/hybridsource/ Rafi On Tue, Oct 26, 2021 at 7:46 PM Qihua Yang wrote: > Hi, > > My flink app has two data sources. One is from a Kafka topic, one

Flink handle both kafka source and db source

2021-10-26 Thread Qihua Yang
Hi, My flink app has two data sources. One is from a Kafka topic, one is from a database by using the JDBC connector. Flink scan the full database table. Which mode should we use? batch mode or streaming mode? How do we know the database table is fully scanned? Will Flink throw any signal to show