Thanks Luke. I wasn’t sure if that was advised or even possible, but it seems
to be working well.
I would like to introduce a 5 minute delay between reading each message and
publishing it to the output topic. Is the correct way to do this in Apache
Beam, using triggers?
I am trying the approach below in my pipeline, but the messages just output
without the delay.
PCollection windowed_inputEvents = inputEvents.apply(
Window.into(FixedWindows.of(Duration.standardMinutes(1)))
.triggering(AfterProcessingTime.pastFirstElementInPane().
plusDelayOf(Duration.standardMinutes(5))).
withAllowedLateness(Duration.standardMinutes(1)).
discardingFiredPanes());
Thanks
> On 25 Mar 2020, at 23:11, Luke Cwik <[email protected]> wrote:
>
> Is there a reason you don't keep them separate and apply the same transforms
> to each output with the only difference being the final writing transform
> being configured for the correct topic?
>
> On Wed, Mar 25, 2020 at 1:54 PM Andrew Wylie <[email protected]
> <mailto:[email protected]>> wrote:
> Hi,
>
> So I have 5 input Pub Sub topics and each one of those has an output Pub Sub
> topic. When I Flatten the 5 inputs, is there a way I can ensure that I can
> route the messages to their respective output topics? Is there any kind of
> built in attribute in the messages I can read to determine their input topic
> and therefore which output topic to write to?
>
> Pub Sub Input topic 1 messages need to be output from the pipeline to Pub Sub
> Output topic 1 for example.
>
>
>
> Thanks
>
