If a transform doesn't return something waitable, there is no way to wait on it. However: * AvroIO.write is waitable - if not through AvroIO.write() (I don't remember off the top of my head), then at least through FileIO.write().via(AvroIO.sink()). * KinesisIO.write is *very* easy to change to be waitable - as you can see here <https://github.com/apache/beam/blob/master/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisIO.java#L705>, it simply applies a ParDo. Merely returning the result of that, instead of returning PDone.in(), would make it waitable.
On Sat, Jun 27, 2020 at 3:39 AM Mani Kolbe <[email protected]> wrote: > I am using sparkRunner on streaming mode. So stateful DoFn is not > supported. > > On Sat, 27 Jun, 2020, 7:48 AM Sunny, Mani Kolbe, <[email protected]> wrote: > >> Hello, >> >> >> >> I am looking to implement Wait.on() pattern to do something after writes >> for each window are done. There are two outputs - one writing using AvroIO >> and other using KinesisIO. The problem is writes don’t return PCollection >> which is required to construct Wait.On(). Is there a way around? >> >> >> >> Regards, >> >> Mani >> >
