Just to organize my thoughts and to refresh myself on how this would integrate with each SDK, I wrote this short follow-up on the API, PaneInfo encoding, and proto changes. I'm especially interested in feedback from people familiar with various SDKs about mistakes I've made in that SDK and the most idiomatic ways to represent this.
https://s.apache.org/beam-drain-mode Kenn On Tue, Mar 18, 2025 at 11:42 AM Kenneth Knowles <k...@apache.org> wrote: > Thanks for all the detailed insightful comments! I've made some edits to > reflect your insights and also settled on what I think is the most feasible > plan, considering effort and backwards-compatibility. > > The work is actually now quite small (yay!). I've added these at the > bottom of the doc: > > - Add "is drain" bit for aggregations > - Add "is drain" bit for timer callbacks > - Fire processing time timers instantly during drain (likely getting > dropped because their window is expired) > - Immediately drop already-expired processing time timers (since they are > doomed to be dropped when they fire) > > This is a set of backwards-compatible changes consistent with "option 1" > change proposal in the doc. > > Let me know if you think this is not enough or too much. > > Kenn > > On Wed, Mar 12, 2025 at 6:33 PM Jan Lukavský <je...@seznam.cz> wrote: > >> Hi Kenn, >> >> thanks for putting this down on paper. This is great initiative as it >> might touch some core parts of the model we are actually somewhat circling >> around. I left some comments and I'm looking forward to the broad >> discussion this definitely deserves. >> >> Jan >> On 3/11/25 15:46, Kenneth Knowles wrote: >> >> Hi all, >> >> I've spent some time on the rather hairy details of timers, batch, drain, >> and timer loops lately. We have some inconsistencies in this area that >> regularly bite users. >> >> I've written up what I *think* are the desired semantics. some notes >> about the current status of implementation, and a few proposals for how we >> can improve things. >> >> https://s.apache.org/beam-timers-and-drain >> >> Please take a look! This stuff is hard and I could really use as many >> smart pairs of eyes on this as possible. >> >> Kenn >> >>
