On Wed, Jul 26, 2017 at 9:43 PM, Kenneth Knowles <k...@google.com.invalid> wrote:
> This is a bit of an improvised change to the Beam model, if these are > really treated *that* specially. (notably, they are a subset of the > WindowFns that we ship with our SDKs, so it really is a careful selection) > > It does make sense to have some special WindowFns with distinguished > semantics, since a lot of use cases are covered without the generality of > the Beam model. Having a compact proto repr for these makes sense, but the > particular choice of functions I had viewed as a hack while the Fn API was > maturing to support the real model. > Compactness of the proto representation isn't important, but it is good to often have a language-independent representation (and definition). Many common UserFns should likely have a language-independent urn and format, especially once we start looking at multi-language pipelines. If this is a new special category in the model (capability matrix?) there > should be a lot more input and deliberation as to what they are. TBH I > think we will probably choose just the WindowFns that you did, but the > standardization deserves attention and documentation. > These are already called out individually in the compatibility matrix, which probably makes sense as it allows a runner to declare "partial" support for windowing. On Wed, Jul 26, 2017 at 9:34 PM, Robert Bradshaw < > rober...@google.com.invalid> wrote: > > > I think there may be a distinction between hard-coding support for the > > "standard" WindowFns (e.g. > > https://github.com/apache/beam/blob/master/sdks/common/ > > runner-api/src/main/proto/standard_window_fns.proto) > > and accepting WindowFns as a UDF. Different runners have offered > different > > levels of support for this in the past (for example, the Fn API doesn't > > support WindowFns other than these standard ones unless they're > implemented > > in Java--and even then it'd probably be better for these to be executed > in > > the SDK context). > > > > On Wed, Jul 26, 2017 at 9:16 PM, Kenneth Knowles <k...@google.com.invalid > > > > wrote: > > > > > Hi Etienne, > > > > > > Every WindowFn is a UDF, so there is really no such thing as "custom" > > > window merging. Is this the same as saying that a runner supports only > > > merging for Sessions? Or just supports WindowFn that merges based on > > > overlap? > > > > > > Kenn > > > > > > On Mon, Jul 24, 2017 at 10:15 AM, Etienne Chauchot < > echauc...@gmail.com> > > > wrote: > > > > > > > Hi all, > > > > > > > > There is now 2 new ValidatesRunner tests: WindowTest. > > > testMergingCustomWindows > > > > and WindowTest.testMergingCustomWindowsKeyedCollection. The aim of > > these > > > > tests is to verify that the runners can handle custom windowFn > > > (extensions > > > > of windowFn that, for example, could rely on elements in addition to > > > > timestamps). > > > > > > > > As new runners are coming, I wanted to let you know that there is > also > > a > > > > new category tag UsesCustomWindowMerging that you can use to skip > these > > > > tests while running ValidatesRunner tests on runners that do not > > support > > > > custom window merging yet. > > > > > > > > Besides, there is also an ongoing related PR ( > > > > https://github.com/apache/beam/pull/3592) to enhance the test utils > > > > methods of WindowFnTestUtils. > > > > > > > > Etienne > > > > > > > > > > > > > > > > > > > > > > > > > >
