Hi Robert, Could you please clarify how session / merging windows are supported for Python pipelines? The SDK has it, which runner supports it and what limitations are there, if any?
Thanks, Thomas On Tue, Apr 3, 2018 at 10:41 AM, Robert Bradshaw <[email protected]> wrote: > On Tue, Apr 3, 2018 at 10:26 AM Thomas Weise <[email protected]> wrote: > >> >> On Mon, Apr 2, 2018 at 9:55 PM, Ahmet Altay <[email protected]> wrote: >> >>> >>>> I’m specifically interested in stateful processing and timers as basic >>>> building blocks that could be used with a global window to implement >>>> session functionality and other higher level abstractions without being >>>> constrained by window functions and predefined triggers. >>>> >>> > While we certainly want to fill this in, I am curious what processing > you're doing that's not satisfied by the standard window functions > (including sessions) and triggers. > > >> >>> These are missing. Charles started working on those [1]. We can use any >>> help we can get if you are interested. >>> >> >> Are there any existing discussions or docs for BEAM-2687 >> <https://issues.apache.org/jira/browse/BEAM-2687>? >> > > There are some sketches at https://docs.google.com/document/d/ > 1ClmQ6LqdnfseRzeSw3SL68DAO1f8jsWBL2FfzWErlbw/edit#heading=h.pv99fae1rece > , but it would be worth fleshing this out into a full proposal. > > >> >> >>> >>>> >>>> Also since we have the runner capability matrix >>>> <https://beam.apache.org/documentation/runners/capability-matrix/>, >>>> would it be useful to track SDK capabilities in a similar way so that >>>> users know what’s supported? >>>> >>> >>> I think this would be really helpful. Especially now that we have >>> multiple SDKs in master. Recently Rafael proposed having per-transform >>> documentation [2]. Building an SDK capability matrix would be natural >>> extension of it. >>> >> >> To have some basic orientation, it might be good add the Python SDK to >> the existing matrix? First column currently is "Beam Model", should that >> become Java SDK instead? >> > > Here again it depends on the runner. By this matrix, the Python SDK has > the same coverage as "Beam Model" except for "Stateful Processing" and a ~ > for Side Inputs (batch support, streaming is a work in progress) for > local/dataflow. Both are less complete for Portability-based runners. > > >
