I've considered a similar question before: Dynamic addition of Sinks /
based on some external configuration.
The answer I've mostly been given is: this is a bad idea. The checkpoint
state that flink uses for job recovery is dependent on the topology of the
job, and dynamically adding more sinks changes this topology.
You might be able to get away with dynamically submitting multiple jobs in
a single Flink application, but you'd have to restart the application every
time you add a new job.
I've not tried this though, so hopefully someone smarter can come in and
advise as well :)
-Andrew Otto
Wikimedia Foundation
On Mon, Jun 5, 2023 at 8:27 AM Yogesh Rao wrote:
> Hello,
>
> I am trying out flink for one stream processing scenario and was wondering
> if it can be achieved using Apache Flink. So any pointers regarding how it
> can be achieved will be of great help.
>
> Scenario :-
>
> A kafka topic has the input for stream processing, multiple applications
> lets say A & B would be publishing their message to the same topic (Topic
> X) with different keys (keys being application names). These messages are
> read by stream processing applications and processed eventually landing in
> sinks specific for A & B. The end result is to have this entire piece
> dynamic so that new applications C,D,E etc.. can be automatically
> accommodated.
>
> ATM i am able to figure out the kafka source and stream processing part.
> What I am not clear is incase of streaming would conditional multiple sinks
> work ? i.e. for Application A data lands into Sink A, Application B -> Sink
> B and so on .
>
> From Implementation I could probably split the stream and pass those
> streams to respective tables. However all this needs to happen dynamically.
>
> Would Apache Flink be able to support this ? if yes how?
>
> I am using Apache Flink 1.17.1 with the pipeline written in Java
>
> Thank you in advance,
>
> Regards,
> -Yogesh
>
