@Kenn, 1. when the watermark jumps from 0 to 7, http://foo -> http://foo/bar -> http://foo will be emitted We can emit events with timestamps before watermark in the pane 2. http://foo -> http://foo/bizzle -> http://foo/bar -> http://foo will be emitted if it's within the allowed lateness which Beam already allows us to do.
To elaborate on the use case, when users are visiting Amazon we want to offer them best recommendations. Thus, we would like to know what leads to their final decision and track the pages they visit until clicking the "Add to cart" button. It will be too late if we only send the results when they finish shopping. @Tyler, I don't think it's likely to happen for my use case. Think about a user jumping between pages like crazy. Meanwhile, we can control how fast watermark progresses as long as it meets the latency requirement. On Fri, Dec 2, 2016 at 11:45 AM Tyler Akidau <taki...@google.com> wrote: > And one more question while we're at it: what if you have events happening > every second within the window? Do you really want to emit a new pane every > second as the watermark progresses (assuming it progresses relatively > smoothly)? What if we're talking differences of event times of > milliseconds? Is one pane per millisecond what you want? > > -Tyler > > On Fri, Dec 2, 2016 at 10:41 AM Kenneth Knowles <k...@google.com> wrote: > > Thanks for laying out some details. > > On Thu, Dec 1, 2016 at 7:09 PM, Manu Zhang <owenzhang1...@gmail.com> > wrote: > > Yes, the difficulty is to define that trigger. The existing triggers fire > at the end of window. (I could be mistaken, which will be good news) > > > You are not mistaken that the only existing event time trigger is the one > that fires at the end of the window. The trigger you describe would be a > new primitive trigger. It fits with the design, if we ensure monotonicity, > etc. Actually implementing it in the backend is easy, of course. We > actually had something like it, but didn't quite nail it down so we removed > it until we had a solid use case and design for it. > > B and C which are not mutually exclusive > More on my use case. Say a user visits http://foo at 1, http://foo/bar at > 4 and back to http://foo at 5 all in a Session > we would want to emit > > http://foo when the watermark passes 1 > http://foo -> http://foo/bar when the watermark passes 4 > http://foo -> http://foo/bar -> http://foo when the watermark passes 5 > > > What would you want to emit when the watermark jumps from 0 to 7 and all > three of the above are buffered? > > What would you want to emit when the watermark was at 9 and > http://foo/bizzle came in with timestamp 3? > > Kenn > > > > > On Fri, Dec 2, 2016 at 10:12 AM Ben Chambers <bchamb...@google.com> wrote: > > As a clarifying question: > > If you have three elements in the pane with timestamps [1, 4, 5], would > you: > A. want to emit that entire pane when the watermark passes 1 > B. want to emit that entire pane when the watermark passes 5 > C. emit a fragment of that pane containing only the first element when the > watermark passes 1 > > On Thu, Dec 1, 2016 at 6:01 PM Tyler Akidau <taki...@google.com> wrote: > > So what you want is essentially a trigger that fires when the watermark > has passed the event time of the oldest un-emitted element in the current > pane? You could them presumably wrap this in a repeat to get the overall > desired semantics, right? > > -Tyler > > > On Fri, Dec 2, 2016 at 7:32 AM Manu Zhang <owenzhang1...@gmail.com> wrote: > > My use case is to track user trajectory based on page view event when they > visit a website. The input would be like a list of PageView(userId, url, > eventTimestamp) with watermarks (= eventTimestamp - duration). I'm trying > Sessions with event time trigger. Note we can't wait for the end of session > window due to latency requirement. Instead, we want to emit the user > trajectories whenever a buffered PageView's event time is passed by > watermark. > > On Fri, Dec 2, 2016 at 5:41 AM Lukasz Cwik <lc...@google.com> wrote: > > Can you provide more details about the problem your trying to solve with > some examples showing input and the expected output? > > > > > On Wed, Nov 30, 2016 at 11:08 PM, Manu Zhang <owenzhang1...@gmail.com> > wrote: > > Hi, > > Recently I’m addressing a problem where users want to trigger after > watermark past each element (i.e. in the middle of event-time window). I > fail to find an existing trigger that does so. Any idea on model this > problem with Beam ? > > Thanks, > Manu Zhang > > >
