je-ik commented on issue #23379: URL: https://github.com/apache/beam/issues/23379#issuecomment-1263253868
Do we have these definitions "set in stone"? I'd offer an alternative definition of watermark - not actually being an aggregation of "elements", it is rather an aggregation of causality. I tried to explain it [here](https://twitter.com/janl_apache/status/1478757956263071745?s=20&t=4SarwNn0_gNXd35FlK-pYg). The basic idea is that the invariant is not the actual value of watermark is what is important. The important part is that causality must be preserved. If event B is caused by event B in any upstream partition of any transform (e.g. source), then the same order of these events (that is, first is observed event A and only then event B) must be preserved in *all* downstream partitions of *all* transforms and this must hold even in the presence of failures. With this definition we can view watermark as an "event-time wall clock", This wall clock may move at any non-negative speed wrt to processing-time. This definition also gives sense to "last open window", because the "window close" is simple an event as any other, having assigned timestamp that can be compared to this wall-clock. Would this definition brake any concepts we currently use? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
