There is no need for a trigger here at all. The stateful DoFn will process elements as they arrive, so you don't need to set any triggering.
Also, have you seen the GroupIntoBaches transform? It may already do what you are trying to do. On Thu, Jan 28, 2021 at 8:00 PM Pradyumna Achar <[email protected]> wrote: > Hello, > > I am running into a strange situation trying to use windows and FileIO > properly. > > I have a KafkaIO source, followed by a DoFn that assigns timestamps based > on a field in the input record using outputWithTimestamp. After that, I > apply FixedWindows of 1 hour duration on these elements. I need to write > these windows to disk in parts, with a constraint that each part be of a > certain size (except the last part). > > So, I made the FixedWindow trigger repeatedly, once per element, and > implemented a stateful DoFn that collects these elements until the size > limit is reached and outputs an Iterable. > > However, now I see that these elements are in separate panes. FileIO's > behavior is that "writing happens by default per window and pane" (per > javadoc), and this is what I am observing too. I get a bunch of files, > instead of one for the Iterable. > > Is there any way I can make FileIO write that Iterable to a single file? > > Thank you > - Pradyumna >
