Hi Sunny, Welcome to Flink đ.
The next thing for you to consider is to setup checkpointing [1] which allows a failing job to pick up from where it stopped. Sincere greetings from the supposed close-by Zurich đ Thias [1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/checkpointing/ From: Raihan Sunny <raihan.su...@selise.ch> Sent: Tuesday, December 20, 2022 6:30 AM To: user@flink.apache.org Subject: Understanding pipelined regions â EXTERNAL MESSAGE â CAUTION: Think Before You Click â Hi, I'm quite new to the world of stream and batch processing. I've been reading about pipelined regions in Flink and am quite confused by what it means. My specific problem involves a streaming job that looks like the following: 1. There is a Kafka source that takes in an input data that sets off a series of operations 2. As part of the first operation, I have an operator that produces multiple values, each of which has to be fed into several different operators in parallel 3. The operators each produce a result which I keyBy and merge together using the union operator 4. The merged result is then written to a Kafka sink The problem is that when one of the parallel operators throws an exception, all the tasks in the entire pipeline gets restarted including the source which then replays the input data and the process starts off once again. My question is if it's possible to make the tasks of only the branch that failed restart rather than the whole job. I do realize that it is possible to split up the job such that the first operator produces its output to a sink and having that as the source to the subsequent operations can mitigate the problem. I was just wondering if it's possible in the scenario that I have described above. In general, how can I "create" a pipelined region? Thanks, Sunny [Image removed by sender.] Secure Link Services Group ZĂŒrich: The Circle 37, 8058 ZĂŒrich-Airport, Switzerland Munich: Tal 44, 80331 MĂŒnchen, Germany Dubai: Building 3, 3rd Floor, Dubai Design District, Dubai, United Arab Emirates Dhaka: Midas Center, Road 16, Dhanmondi, Dhaka 1209, Bangladesh Thimphu: Bhutan Innovation Tech Center, Babesa, P.O. Box 633, Thimphu, Bhutan Visit us: www.selise.ch<http://www.selise.ch> Important Note: This e-mail and any attachment are confidential and may contain trade secrets and may well also be legally privileged or otherwise protected from disclosure. If you have received it in error, you are on notice of its status. Please notify us immediately by reply e-mail and then delete this e-mail and any attachment from your system. If you are not the intended recipient please understand that you must not copy this e-mail or any attachment or disclose the contents to any other person. Thank you for your cooperation. Diese Nachricht ist ausschliesslich fĂŒr den Adressaten bestimmt und beinhaltet unter UmstĂ€nden vertrauliche Mitteilungen. Da die Vertraulichkeit von e-Mail-Nachrichten nicht gewĂ€hrleistet werden kann, ĂŒbernehmen wir keine Haftung fĂŒr die GewĂ€hrung der Vertraulichkeit und Unversehrtheit dieser Mitteilung. Bei irrtĂŒmlicher Zustellung bitten wir Sie um Benachrichtigung per e-Mail und um Löschung dieser Nachricht sowie eventueller AnhĂ€nge. Jegliche unberechtigte Verwendung oder Verbreitung dieser Informationen ist streng verboten. This message is intended only for the named recipient and may contain confidential or privileged information. As the confidentiality of email communication cannot be guaranteed, we do not accept any responsibility for the confidentiality and the intactness of this message. If you have received it in error, please advise the sender by return e-mail and delete this message and any attachments. Any unauthorised use or dissemination of this information is strictly prohibited.
