created BEAM-1119 <https://issues.apache.org/jira/browse/BEAM-1119> and assigned to you.
On Fri, Dec 9, 2016 at 11:51 AM Kenneth Knowles <[email protected]> wrote: > I don't think there is a JIRA for this, but I agree that the use case is > good and compatible with the model. Please do open a JIRA ticket in the > beam-model component. Feel free to assign to me or leave unassigned. > > On Thu, Dec 8, 2016 at 5:30 PM, Manu Zhang <[email protected]> > wrote: > > @Kenn and @Tyler, > > Given the use case is defined (hope I've explained it clearly), do we have > plans/jiras to add the new functionality ? > > Thanks, > Manu > > On Fri, Dec 2, 2016 at 12:21 PM Manu Zhang <[email protected]> > wrote: > > @Kenn, > > 1. when the watermark jumps from 0 to 7, http://foo -> http://foo/bar -> > http://foo will be emitted > We can emit events with timestamps before watermark in the pane > 2. http://foo -> http://foo/bizzle -> http://foo/bar -> http://foo will > be emitted if it's within the allowed lateness > which Beam already allows us to do. > > To elaborate on the use case, when users are visiting Amazon we want to > offer them best recommendations. > Thus, we would like to know what leads to their final decision and track > the pages they visit until clicking the "Add to cart" button. > It will be too late if we only send the results when they finish shopping. > > @Tyler, > I don't think it's likely to happen for my use case. Think about a user > jumping between pages like crazy. Meanwhile, we can control how fast > watermark progresses as long as it meets the latency requirement. > > > > > On Fri, Dec 2, 2016 at 11:45 AM Tyler Akidau <[email protected]> wrote: > > And one more question while we're at it: what if you have events happening > every second within the window? Do you really want to emit a new pane every > second as the watermark progresses (assuming it progresses relatively > smoothly)? What if we're talking differences of event times of > milliseconds? Is one pane per millisecond what you want? > > -Tyler > > On Fri, Dec 2, 2016 at 10:41 AM Kenneth Knowles <[email protected]> wrote: > > Thanks for laying out some details. > > On Thu, Dec 1, 2016 at 7:09 PM, Manu Zhang <[email protected]> > wrote: > > Yes, the difficulty is to define that trigger. The existing triggers fire > at the end of window. (I could be mistaken, which will be good news) > > > You are not mistaken that the only existing event time trigger is the one > that fires at the end of the window. The trigger you describe would be a > new primitive trigger. It fits with the design, if we ensure monotonicity, > etc. Actually implementing it in the backend is easy, of course. We > actually had something like it, but didn't quite nail it down so we removed > it until we had a solid use case and design for it. > > B and C which are not mutually exclusive > More on my use case. Say a user visits http://foo at 1, http://foo/bar at > 4 and back to http://foo at 5 all in a Session > we would want to emit > > http://foo when the watermark passes 1 > http://foo -> http://foo/bar when the watermark passes 4 > http://foo -> http://foo/bar -> http://foo when the watermark passes 5 > > > What would you want to emit when the watermark jumps from 0 to 7 and all > three of the above are buffered? > > What would you want to emit when the watermark was at 9 and > http://foo/bizzle came in with timestamp 3? > > Kenn > > > > > On Fri, Dec 2, 2016 at 10:12 AM Ben Chambers <[email protected]> wrote: > > As a clarifying question: > > If you have three elements in the pane with timestamps [1, 4, 5], would > you: > A. want to emit that entire pane when the watermark passes 1 > B. want to emit that entire pane when the watermark passes 5 > C. emit a fragment of that pane containing only the first element when the > watermark passes 1 > > On Thu, Dec 1, 2016 at 6:01 PM Tyler Akidau <[email protected]> wrote: > > So what you want is essentially a trigger that fires when the watermark > has passed the event time of the oldest un-emitted element in the current > pane? You could them presumably wrap this in a repeat to get the overall > desired semantics, right? > > -Tyler > > > On Fri, Dec 2, 2016 at 7:32 AM Manu Zhang <[email protected]> wrote: > > My use case is to track user trajectory based on page view event when they > visit a website. The input would be like a list of PageView(userId, url, > eventTimestamp) with watermarks (= eventTimestamp - duration). I'm trying > Sessions with event time trigger. Note we can't wait for the end of session > window due to latency requirement. Instead, we want to emit the user > trajectories whenever a buffered PageView's event time is passed by > watermark. > > On Fri, Dec 2, 2016 at 5:41 AM Lukasz Cwik <[email protected]> wrote: > > Can you provide more details about the problem your trying to solve with > some examples showing input and the expected output? > > > > > On Wed, Nov 30, 2016 at 11:08 PM, Manu Zhang <[email protected]> > wrote: > > Hi, > > Recently I’m addressing a problem where users want to trigger after > watermark past each element (i.e. in the middle of event-time window). I > fail to find an existing trigger that does so. Any idea on model this > problem with Beam ? > > Thanks, > Manu Zhang > > > >
