Hi Yanfei, Thanks for your proposal! The FLIP contains a lot of great new ideas. I'd like to ask some questions to make sure we are on the same page.
> For the asynchronous interface, Record A should run with Read, Update and Output, while Record B should stay at the Blocking buffer. 1. With this example, does it mean that there will be three mails for Read, Update, and Output ? 2. If yes, since the Read, Update, and Output have to be executed before Record B, does it make sense to encapsulate one mail instead of 3 mails with more overhead? There must be some thoughts behind the design. Look forward to it. > The challenge arises in determining when all the processing logic associated with Record A is fully executed. To address this, we have adopted a reference counting mechanism that tracks ongoing operations (either processing input or executing a callback) related to a single record. The design reminds me of the JVM reference counting for GC. Would you like to add more description for cases when exceptions happened? E.g. when reading or/and updating State throws IOExceptions. > In more detail, AEC uses a inFilghtReocordNum variable to trace the current number of records in progress. Every time the AEC receives a new record, the inFilghtReocordNum increases by 1; when all processing and callback for this record have completed, the inFilghtReocordNum decreases by 1. When processing one checkpoint mail, the current task thread will give up the time slice through the yield() method of the mailbox executor, so that the ongoing state request’s callback and the blocking state requests will be drained first until inFlightRecordNum reduces to 0. 1. Speaking of draining, is it correct to understand that AEC is stateless? E.g. AEC could be easily scaled out if it became a bottleneck. 2. There are Pseudo-code for the inFilghtReocordNum increment, would you like to add Pseudo-code for the inFilghtReocordNum decrement to help us understand the logic better? The FLIP shows overall a great design! +1 for it! Looking forward to your thoughts, thanks! Best regards, Jing On Thu, Mar 7, 2024 at 10:05 AM Yanfei Lei <fredia...@gmail.com> wrote: > Hi devs, > > I'd like to start a discussion on FLIP-425: Asynchronous Execution > Model[1], which is a sub-FLIP of FLIP-423: Disaggregated State Storage > and Management[2]. > > FLIP-425 introduces a non-blocking execution model leveraging the > asynchronous APIs introduced in FLIP-424[3]. > For the whole story please read the FLIP-423[2], and this thread is > aimed to discuss the details of "FLIP-425: Asynchronous Execution > Model". > > Regarding the details of this FLIP, there have been some discussions > here[4], mainly focusing on framework overhead profiling, watermark > processing, etc. Please see link[4] for the context. > > Looking forward to hearing from you! > > > [1] https://cwiki.apache.org/confluence/x/S4p3EQ > [2] https://cwiki.apache.org/confluence/x/R4p3EQ > [3] https://cwiki.apache.org/confluence/x/SYp3EQ > [4] https://lists.apache.org/thread/ct8smn6g9y0b8730z7rp9zfpnwmj8vf0 > > Best, > Yanfei >
