Hi devs, I'd like to start a discussion on FLIP-519: Introduce async lookup key ordered mode[1].
The Flink system currently supports both record-level ordered and unordered output modes for asynchronous lookup joins. However, it does not guarantee the processing order of records sharing the same key. As highlighted in [2], there are two key requirements for enhancing async io operations: 1. Ensuring the processing order of records with the same key is a common requirement in DataStream. 2. Sequential processing of records sharing the same upsertKey when performing lookup join in Flink SQL is essential for maintaining correctness. This optimization aims to balance correctness and performance for stateful streaming workloads.Then the FLIP introduce a new operator KeyedAsyncWaitOperator to supports the optimization. Besides, a new option is added to control the behaviour avoid influencing existing jobs. please find more details in the FLIP wiki document[1]. Looking forward to your feedback. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-519%3A++Introduce+async+lookup+key+ordered+mode [2] https://lists.apache.org/thread/wczzjhw8g0jcbs8lw2jhtrkw858cmx5n Best, Xu Shuai
