Hi Piotrek, Thank you for replying! I want to process each record as soon as it is ingested (or reaches an operator) without waiting for a window for records to arrive. However, by not using windows, I am not sure if each record gets emitted immediately upon processing.
> You still can use windowing, but you might want to emit updated value of the window per every processed record. How do I do this? Also, is there any way I can change the execution.buffer-timeout or setbuffertimeout(milliseconds) dynamically while the job is running? Thank you, Pankaj On Wed, Oct 14, 2020 at 9:42 AM Piotr Nowojski <[email protected]> wrote: > Hi Pankaj, > > I'm not entirely sure if I understand your question. > > If you want to minimize latency, you should avoid using windows or any > other operators, that are buffering data for long periods of time. You > still can use windowing, but you might want to emit updated value of the > window per every processed record. > > Other than that, you might also want to reduce `execution.buffer-timeout` > from the default value of 100ms down to 1ms, or 0ms [1]. Is this what you > are looking for? > > Piotrek > > [1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html > > śr., 14 paź 2020 o 12:38 Pankaj Chand <[email protected]> > napisał(a): > >> Hi all, >> >> What is the recommended way to make a Flink job that processes each event >> individually as soon as it comes and without waiting for a window, in order >> to minimize latency in the entire DAG of operators? >> >> For example, here is some sample WordCount code (without windws), >> followed by some known ways: >> >> DataStream<WordWithCount> wordCounts = text >> .flatMap(new FlatMapFunction<String, WordWithCount>()) >> .keyBy("word") >> .reduce(new ReduceFunction<WordWithCount>()); >> >> >> 1. Don't include any TimeWindow/CountWindow function (does this >> actually achieve what I want?) >> 2. Use a CountWindow with a count of 1 >> 3. Make a Trigger that fires to process each event when it comes in >> >> I think the above methods only work at the processing level and latency >> with respect to a single operator, but does not affect the latency of an >> event in the entire Flink job's DAG since those ways do not affect the >> buffertimeout value. >> >> Thanks, >> >> Pankaj >> >
