danny0405 opened a new pull request, #20777: URL: https://github.com/apache/hudi/pull/20777
### Describe the issue this Pull Request addresses With Flink async lookup joins enabled, each Hudi lookup function can start up to 16 worker threads by default. Every worker acquires the same monitor before calling the wrapped function, so only one lookup can access the cache at a time. Concurrent submissions can leave several workers waiting while one reads or refreshes the cache. The async wrapper was introduced in [#18193](https://github.com/apache/hudi/pull/18193). ### Summary and Changelog Create a single-thread executor when the wrapper opens and submit each lookup to it. The executor serializes access to the wrapped function and keeps lookup work off the Flink mailbox thread. Remove the lookup monitor and the worker-count argument from the wrapper constructor and provider factory. ### Impact Jobs using `lookup.async` use one worker per lookup function. Existing configurations can still specify `lookup.async-thread-number`. The option is deprecated and its value is ignored. Multiple lookup futures can still be outstanding. ### Risk Level low The change affects executor lifecycle and scheduling for async lookups. The wrapper test checks that successive lookups run on the same worker, distinct from the caller. It also covers exception propagation. The provider test covers synchronous and asynchronous provider selection. #### Testing Ran `TestAsyncLookupFunctionWrapper` and `TestLookupUtilities` with the Flink 1.20 Maven profile. All four tests passed. ### Documentation Update Updated the Flink option description and default to document the single worker and deprecation of `lookup.async-thread-number`. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
