llo,
>
> We did end up opening a GCP ticket. This is the advice they gave us:
>
> "I understand that you are getting an error message [1] when working with
> a custom windowing function. You are trying the options as
>
> “bounded session” and “ per key fixed window” as men
Hello,
We did end up opening a GCP ticket. This is the advice they gave us:
"I understand that you are getting an error message [1] when working with a
custom windowing function. You are trying the options as
“bounded session” and “ per key fixed window” as mentioned from the third
Sundhar
Technical Solutions Engineer, Data
s vetaksund...@google.com
On Fri, Mar 17, 2023 at 1:02 PM Sahith Nallapareddy via dev <
dev@beam.apache.org> wrote:
> Hello,
>
> We are working on writing a custom windowing function. The functionality
> is similar to the one desc
Hello,
We are working on writing a custom windowing function. The functionality is
similar to the one described in this book
https://www.oreilly.com/library/view/streaming-systems/9781491983867/ch04.html
(the bounded session and per key fixed window is what we are trying).
However, we
I'm not sure if it's currently legal. However the watermark is generally
defined to be monotonic, so if it was allowed it would result in late data
in the pipeline.
On Tue, Nov 5, 2019 at 10:29 AM Aaron Dixon wrote:
> Thanks Reuven,
>
> So is my conclusion correct? That it is illegal for any
Thanks Reuven,
So is my conclusion correct? That it is illegal for any custom window
function (+ combiner policy) to merge in a way that would regress the
watermark?
What do Runners (eg Dataflow) do if this occurs?
Does the API obligate runners to fail, or can insanity ensue? :)
On Tue, Nov 5,
On Tue, Nov 5, 2019 at 8:07 AM Aaron Dixon wrote:
> I noticed that if I use TimestampCombiner/EARLIEST for session windows
> that the watermark appears to get held up for sessions that never "close"
> (or that extend for a long time).
>
Correct - because the watermark is then being held up by
I noticed that if I use TimestampCombiner/EARLIEST for session windows that
the watermark appears to get held up for sessions that never "close" (or
that extend for a long time).
But if I use default (TimestampCombiner/END_OF_WINDOW) the watermark
doesn't get held.
Does this mean that the
