Hi, I have an API that emits output that I want to use as a data source for Flink.
I have written a custom source function that is as follows - public class DynamicRuleSource extends AlertingRuleSource { private ArrayList<Rule> rules = new ArrayList<Rule>(); public void run(SourceContext<Rule> ctx) throws Exception { System.out.println("In run "); while(true) { while (!rules.isEmpty()) { Rule rule = rules.remove(0); ctx.collectWithTimestamp(rule, 0); ctx.emitWatermark(new Watermark(0)); } Thread.sleep(1000); } } public void addRule(Rule rule) { rules.add(rule); } @Override public void cancel() { } } When the API is invoked, it calls the addRule method in my CustomSource function. The run method in CustomSource polls for any data to be ingested. The same object instance is shared with the API and the Flink Execution environment, however, the output of the API does not get ingested into the Flink DataStream. Is this the right pattern to use, or is Kafka the recommended way of streaming data into Flink ? --Aarti -- Aarti Gupta <https://www.linkedin.com/company/qualys> Director, Engineering, Correlation aagu...@qualys.com T Qualys, Inc. – Blog <https://qualys.com/blog> | Community <https://community.qualys.com> | Twitter <https://twitter.com/qualys> <https://www.qualys.com/email-banner>