Hi Kenneth,
Thanks, I can eventually share the code but it needs some work to remove all
references to any sensitive data. The way I have gone about fixing this so far
is executing the code and dealing with the error messages related to
window/trigger.
The two messages which come up are the following ones
Inputs to Flatten had incompatible triggers:
Repeatedly.forever(AfterProcessingTime.pastFirstElementInPane().plusDelayOf(15
seconds)),
Repeatedly.forever(AfterSynchronizedProcessingTime.pastFirstElementInPane())
--
GroupByKey cannot be applied to non-bounded PCollection in the GlobalWindow
without a trigger. Use a Window.into or Window.triggering transform prior to
GroupByKey.
So working my way through the errors I have added the following transform
.apply("win_1", Window.<Row>into(new
GlobalWindows()).triggering(commonWindowTrigger).discardingFiredPanes());
where the common trigger looks like
static final Trigger commonWindowTrigger =
Repeatedly.forever(AfterProcessingTime.pastFirstElementInPane().plusDelayOf(Duration.standardSeconds(15)));
I count six occurrences in the pipeline and removing any one of those
transforms will trigger one of the above mentioned error messages. As it stands
the code does run but it just feels a little messy. (its messy any ways but I
am working on that ;-) )
So going back to my question and your comment, is there no way to setup the
trigger once for the Global Window and it would use that for the rest of the
pipeline ?
Thanks,
Serge
On 26 May 2021 at 18:35:46, Kenneth Knowles ([email protected]) wrote:
When you set up a trigger at the beginning of the pipeline, all the later
GroupByKey/Combine operations should automatically get a trigger that mostly
"let's the data flow". If you can share more about your pipeline - especially
code - then we could probably help more.
Kenn
On Wed, May 26, 2021 at 9:11 AM Sozonoff Serge <[email protected]> wrote:
Hi,
Well not exactly but maybe I can do something with that.
Is there no way to simply assign some sort of Global trigger to the entire
pipeline so do not have to define it for each Collection in question ?
Thanks,
Serge
On 26 May 2021 at 18:01:14, Kenneth Knowles ([email protected]) wrote:
You can use Window.configure() to only set the values you want to change. Is
that what you mean?
Kenn
On Wed, May 26, 2021 at 8:42 AM Sozonoff Serge <[email protected]> wrote:
Hi All,
I find myself having to pepper Window transforms all over my pipeline, I count
about 9 in order to get my pipeline to run. Aside from the class type they
window, all the statements are identical.
A window into new GlobalWindows() using an identical trigger.
Is there a way to change the trigger on the global window so I could avoid this
extra code all over the place ?
I really dont care about the windowing but its imposed since the pipeline is
unbound and there are operations which require the window.
Thanks,
Serge
