You should be able to simply rewindow _after_ the Stateful DoFn. In this window you do want to trigger on every element, as now each element is an iterable.
On Thu, Jan 28, 2021 at 10:20 PM Pradyumna Achar <[email protected]> wrote: > Thank you. I tried it out and it does pass elements to the stateful DoFn > as and when they arrive instead of waiting for the window to finish, the > way you describe. > > However, with this, the FileIO does not write the file until the whole > window is done, even though the stateful DoFn outputs iterables in chunks > of 100MB. It only receives a single ON_TIME pane and writes a big file of > the entire window. > > Wondering if there is a way to tell that each iterable is a separate pane > or something like that.. > >>
