danny0405 opened a new pull request, #20777:
URL: https://github.com/apache/hudi/pull/20777

   ### Describe the issue this Pull Request addresses
   
   With Flink async lookup joins enabled, each Hudi lookup function can start 
up to 16 worker threads by default. Every worker acquires the same monitor 
before calling the wrapped function, so only one lookup can access the cache at 
a time. Concurrent submissions can leave several workers waiting while one 
reads or refreshes the cache. The async wrapper was introduced in 
[#18193](https://github.com/apache/hudi/pull/18193).
   
   ### Summary and Changelog
   
   Create a single-thread executor when the wrapper opens and submit each 
lookup to it. The executor serializes access to the wrapped function and keeps 
lookup work off the Flink mailbox thread. Remove the lookup monitor and the 
worker-count argument from the wrapper constructor and provider factory.
   
   ### Impact
   
   Jobs using `lookup.async` use one worker per lookup function. Existing 
configurations can still specify `lookup.async-thread-number`. The option is 
deprecated and its value is ignored. Multiple lookup futures can still be 
outstanding.
   
   ### Risk Level
   
   low
   
   The change affects executor lifecycle and scheduling for async lookups. The 
wrapper test checks that successive lookups run on the same worker, distinct 
from the caller. It also covers exception propagation. The provider test covers 
synchronous and asynchronous provider selection.
   
   #### Testing
   
   Ran `TestAsyncLookupFunctionWrapper` and `TestLookupUtilities` with the 
Flink 1.20 Maven profile. All four tests passed.
   
   ### Documentation Update
   
   Updated the Flink option description and default to document the single 
worker and deprecation of `lookup.async-thread-number`.
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to