+1 for the feature. But since having multiple watermark tuples within one streaming window is not useful because WindowedOperator currently only processes watermark only at endWindow, how about:
void processTupleForWatermark(Tuple.WindowedTuple<InputT> tuple); // to be called for each input tuple, updates the state of the impl of WatermarkGenerator interface. ControlTuple.Watermark getWatermarkTuple(); // to be called at endWindow. return null if watermark is not available This is actually somewhat related to the separate debate on this list about control tuples being delivered only at streaming window boundary. David On Thu, Nov 24, 2016 at 2:52 AM, Chinmay Kolhatkar <[email protected]> wrote: > Dear Community, > > I'm working on adding support for heuristic watermark in Windowed Operator. > Heuristic watermark give users of WindowedOperator a way to logically > determine whether watermark condition is met or not by inspecting the > tuples received. > This can act as a replacement for or way to work along with Control Tuple > received on control port. > > Here is the approach I'm considering: > > 1. A new interface lets say "HeuristicWatermark" will be added which > extends Component<Context.OperatorContext> > The reason why its extended with Component is then it can follow a > lifecycle. > > 2. This method contains a single method something like this: > > ControlTuple.Watermark processTupleForWatermark( > Tuple.WindowedTuple<InputT> > input); > > 3. Object of this type can optionally be set to AbstractWindowedOperator as > a plugin which identified whether watermark condition has reached. > > 4. If heuristicWatermark is set, processTupleForWatermark will be called > for every received tuple and the method can return the Watermark object if > watermark condition is met OR return null if not so. > > 5. If return value of this method is non-null, then processWatermark method > will be called which sets the nextWatermark value. And then rest of the > watermark processing can continue to happen in endWindow. > > > Please share your opinion on above approach. > > Thanks, > Chinmay. >
