I like the approach, and the doc! I've added some questions/go specific nits and questions.
I'm lifting one question here for information gathering: Do streaming SDFs in Java/Python require event times be emitted with every element, or is it defaulted to past End of GlobalWindow? On Thu, Mar 31, 2022, 9:19 AM Jack McCluskey <[email protected]> wrote: > Looks good, Danny! > > On Wed, Mar 30, 2022 at 8:22 PM Robert Bradshaw <[email protected]> > wrote: > >> Thanks. I didn't look much at the implementation, but the API makes >> sense to me. >> >> On Wed, Mar 30, 2022 at 10:23 AM Danny McCormick >> <[email protected]> wrote: >> > >> > Hey everyone, I put together a design doc for adding Watermark >> Estimation in the Go Sdk and would appreciate any thoughts you have! >> > >> > Watermark estimation allows a DoFn to estimate the lower bound of data >> yet to be emitted so that future steps can perform aggregation without >> waiting for the DoFn to complete. Right now, both Java and Python support >> watermark estimation by allowing users to specify a custom watermark >> estimator that is invoked when an element/restriction pair is split or >> checkpointed to produce a watermark estimate. They also allow users to >> specify custom watermark state that is persisted across splits and >> checkpoints. This document proposes introducing similar DoFn lifecycle >> methods for creating and managing watermark estimators and state in the Go >> Sdk. >> > >> > Please share any feedback here: >> https://docs.google.com/document/d/1DqCYJ-J1YGNelCRIcN5v6BQsZxJB2l5uWBmN4ti--Ew/edit?usp=sharing >> > >> > Thanks, >> > Danny >> >
