Hi Xuyang, Thanks for driving this! I'm happy to see operator implementation on top of the async state. Overall it looks good, I have one minor question:
It seems the only difference between the new operator implementation and the original one is the way of state accessing. Does this mean the state is totally compatible when toggling the `table.exec.async-state.enabled` from operator's point of view? And by this way users can freely migrate their job to a newer minor version and leverage the newly implemented async operator, without state loss, right? Best, Zakelly On Mon, Aug 5, 2024 at 10:40 AM Xuyang <xyzhong...@163.com> wrote: > Hi devs, > > I'd like to initiate a discussion about FLIP-473: Introduce New SQL > Operators Based on Asynchronous State APIs[1]. > > > > > FLIP-424[2] has introduced an asynchronous state architecture to mitigate > I/O bottlenecks in Flink by offloading state > > access to separate threads from the main thread in each task, thereby > optimizing I/O and computational resource utilization. > > Building upon this infrastructure, FLIP-473[1] proposes introducing new > SQL operators based on these asynchronous state APIs. > > Our goal is to enhance throughput and reduce latency in stateful > operations by leveraging the new asynchronous state architecture > > in Flink SQL jobs. > > > > > For more details, please refer to the FLIP-473[1]. Welcome any feedback > and suggestions for improvement. > > > > > There has been POC code[3] related to the Join operator. > > > > > Best！ > Xuyang > > > > > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-473+Introduce+New+SQL+Operators+Based+on+Asynchronous+State+APIs > > [2] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-424%3A+Asynchronous+State+APIs > > [3] > https://github.com/apache/flink/commit/bccace0f4b233e10279c7d95e009ae6aadad5ae8 > > > > > > > > >
