[
https://issues.apache.org/jira/browse/SAMZA-552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14341795#comment-14341795
]
Jay Kreps commented on SAMZA-552:
---------------------------------
[~criccomini] I think your example is just another way of saying that the
output stream is fundamentally a stream of updates with some "best effort"
suppression of duplicate output.
Your stream does have both
"a", NULL
and
"a", "b"
but the interpretation of that stream is "first I thought ('a',NULL) but then I
revised that to ('a','b') once I got more information". So it's not a
duplicate, it's an update.
I agree that this view of windowed aggregates might be confusing but I once I
got it I actually didn't mind it. In some sense emitting the running count is a
very natural fit for streaming and similar to what you as a human would do.
In terms of efficiency I am less concerned. Two weeks is insanely late, I think
4 hours is a little more reasonable. After all with the "punctuation" model you
would be doing 5 minute aggregates and only giving like a few minutes before
you output the result and close the books on that window. That time period is
equivalent to the retention here, since you will have to drop any event outside
that window. Basically the implementation would be that you have a configurable
setting
event.rejection.horizon=2 hours
and anything after that you drop. So for people who want efficiency they would
set that to just a few minutes but for people who want accuracy at all costs
you would set it to a day.
> Tuple or time window semantics in physical operator
> ---------------------------------------------------
>
> Key: SAMZA-552
> URL: https://issues.apache.org/jira/browse/SAMZA-552
> Project: Samza
> Issue Type: Sub-task
> Components: sql
> Affects Versions: 0.9.0
> Reporter: Yi Pan (Data Infrastructure)
> Assignee: Yi Pan (Data Infrastructure)
>
> The discussion is based on how to support tuple and/or time based window
> operators in Samza physical operator layer.
> Here are the few observations:
> # Tuple represents the “physical ordering” of events while time-based window
> has semantic meanings to users
> # Total ordering between tuples are possible within Samza/Kafka given a
> deterministic MessageSelector on all input streams and offsets within each
> stream
> # No matter whether tuple or time is used to measure the window size, the
> window termination condition is needed to close a window to avoid the job to
> be wedged forever
> The following questions have to be answered to fully implement a window
> operator:
> # how to determine that a window is closed and no new tuples will be added?
> ## For tuple based, how do we close the window if messages do not come or get
> delayed?
> ## For time based, how do we close the window if
> ### the messages are not strictly in order w/ the time?
> ### the message w/ timestamp greater than the window boundary does not come
> or gets delayed?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
