One idea that comes to my mind is to convert ProcessFunction1 with a CoProcessFunction[1]. The processElement1() function can send to side-output and process and maintain the business function message as State without emitting it. Then as Arvid mentioned processElement2() can listen on the side output (emitted by processElement1()) and when it receives it, emit the result from the state and clear the state.
[1] https://ci.apache.org/projects/flink/flink-docs-release-1.9/api/java/org/apache/flink/streaming/api/functions/co/CoProcessFunction.html On Thu, Apr 23, 2020 at 7:20 AM Arvid Heise <ar...@ververica.com> wrote: > Hi Kristoff, > > I see a few ways, none of which are perfect. > > The easiest way would be to not use a sink. Instead of outputting into a > side-output, you could tag that element and have a successive asyncIO place > that in RabbitMQ. If that asyncIO is ordered, then you can be sure that all > following events are only processed after the element has been added. Of > course, the downside is that you have to manually implement the > communication with RabbitMQ and lose what Flink already has. This is what > you already sketched out. > > A more complicated approach would be to implement a custom operator with > input selection to replace processFunction2 [1]. Let's call it op2. You > would add the feedback from the sink implicitly, by also consuming from > that MQ queue on op2. Then, processFunction1 would also emit some flag > event on the main output together with the side output. Op2 would block the > input on receiving that flag until it has read the appropriate entry from > the MQ. However, this approach is really complex to implement and input > selection is somewhat based on a best-effort. So before going that route, > I'd do a small POC to see if it fits your needs. > > The best solution, of course, would be to revise your overall > architecture. It's quite smelly in a stream processing job that you need to > halt execution at some point. If you give some more details, I could try to > help. > > [1] > https://ci.apache.org/projects/flink/flink-docs-master/api/java/index.html?org/apache/flink/streaming/api/operators/InputSelectable.html > > On Wed, Apr 15, 2020 at 5:36 PM KristoffSC <krzysiek.chmielew...@gmail.com> > wrote: > >> My point was, that as far as I know, Sinks are "terminating" operators, >> that >> ends the stream like .collect in Java 8 stream API. The don't emit >> elements >> further and I cannot link then in a way: >> >> source - proces - sink - process - sink >> >> Sink function produces DataStreamSink which is used for emitting elements >> from a streaming topology. >> It is not SingleOutputStreamOperator or DataStream that I can use as input >> for next operator. >> >> >> >> -- >> Sent from: >> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ >> > > > -- > > Arvid Heise | Senior Java Developer > > <https://www.ververica.com/> > > Follow us @VervericaData > > -- > > Join Flink Forward <https://flink-forward.org/> - The Apache Flink > Conference > > Stream Processing | Event Driven | Real Time > > -- > > Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany > > -- > Ververica GmbH > Registered at Amtsgericht Charlottenburg: HRB 158244 B > Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji > (Toni) Cheng >
