Hi all! I have a somewhat complicated stateful DoFn that i would like to add an event time timer on. My goal with the timer is to not output anything until sufficient amount of state has been built up in a Global window.
In doing this I realize that the watermark doesn’t seem to progress at all (regardless of the timer) and in Google Dataflow the displayed watermark is just “-“ when clicking on the ParDo(DoFn) node. The DoFn is reading flattened input from a TextIO.watchForNewFiles and KafkaIO. The Flatten element had a watermark set. I have written tests for my DoFn that all pass using TestStream but since I there explicitly set the watermark progression all is fine. What can I do to look into why there is no watermark progression for a specific PTransform? Regards, Vilhelm von Ehrenheim
