Hi Pankaj, I'm not entirely sure if I understand your question.
If you want to minimize latency, you should avoid using windows or any other operators, that are buffering data for long periods of time. You still can use windowing, but you might want to emit updated value of the window per every processed record. Other than that, you might also want to reduce `execution.buffer-timeout` from the default value of 100ms down to 1ms, or 0ms [1]. Is this what you are looking for? Piotrek [1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html śr., 14 paź 2020 o 12:38 Pankaj Chand <pankajchanda...@gmail.com> napisał(a): > Hi all, > > What is the recommended way to make a Flink job that processes each event > individually as soon as it comes and without waiting for a window, in order > to minimize latency in the entire DAG of operators? > > For example, here is some sample WordCount code (without windws), > followed by some known ways: > > DataStream<WordWithCount> wordCounts = text > .flatMap(new FlatMapFunction<String, WordWithCount>()) > .keyBy("word") > .reduce(new ReduceFunction<WordWithCount>()); > > > 1. Don't include any TimeWindow/CountWindow function (does this > actually achieve what I want?) > 2. Use a CountWindow with a count of 1 > 3. Make a Trigger that fires to process each event when it comes in > > I think the above methods only work at the processing level and latency > with respect to a single operator, but does not affect the latency of an > event in the entire Flink job's DAG since those ways do not affect the > buffertimeout value. > > Thanks, > > Pankaj >
