+1 to adding a Pipeline requirement for this, if business logic relies
on a specific feature runner might/might not have, then Pipeline should
be rejected on runners that do not support it. Do we have a list runners
that have or lack this semantics? Just for clarification - sorry my
ignorance, if this has been already described - do we have a description
of the use-cases that drive this effort?
Thanks,
Jan
On 9/24/21 10:58 PM, Robert Bradshaw wrote:
Thanks for writing this up. Rather than just documenting it, should we
have a way of asserting/requesting it (like time sorted inputs) such
that a pipeline author that needs to rely on this property can be
rejected on runners that don't provide it?
On Fri, Sep 24, 2021 at 12:25 PM Kenneth Knowles <[email protected]> wrote:
Took a look. I definitely agree that something like this is useful, and
well-motivated by the use cases you raise.
Kenn
On Thu, Sep 23, 2021 at 4:30 PM Pablo Estrada <[email protected]> wrote:
Hi all,
I've been spending some time thinking about CDC use cases on Beam. One valuable
piece to enable these use cases is to define how Beam deals with ordering of
elements in streaming pipelines.
With that in mind, I wrote a document[1] that proposes a definition of the
ordering semantics supported by most Beam runners, and a pull request [2] with
ValidatesRunner tests and documentation updates.
Would you please review these, add your comments and thoughts, and let me know
if they make sense?
Thanks!
-P.
[1]
https://docs.google.com/document/d/1_7WRJznXlOtWuVaHl_dpy8OZcx_M8BUmeWVA4G0-wEc/edit#
[2] https://github.com/apache/beam/pull/15378
