Hi all, Dong(cc'ed) and I are opening this thread to discuss our proposal to support buffering & flushing the output of operators with idempotent semantics, which has been documented in FLIP-365<https://cwiki.apache.org/confluence/display/FLINK/FLIP-365%3A+Introduce+flush+interval+to+adjust+the+interval+of+emitting+results+with+idempotent+semantics>.
In the pursuit of unifying batch and stream processing, it has been discovered that the batch execution mode provides a significant advantage by allowing only the final result from "rolling" operations such as reduce() or sum() to be emitted, thus reducing the amount of time and resources required by downstream applications. Inspired by this advantage, the proposed solution supports buffering the output of operators in the streaming mode and periodically emitting the final results, much like in batch processing. This approach is designed to help improve the throughput of jobs by reducing the need for downstream applications to process intermediate results in a stream, at the cost of increased latency and state size. Please refer to the FLIP document for more details about the proposed design and implementation. We welcome any feedback and opinions on this proposal. Best regards. Dong and Yunfeng
