yihua opened a new issue, #20264: URL: https://github.com/apache/hudi/issues/20264
### Task Description **What needs to be done:** Check whether the timeline server view is behind the client without taking the per-table view lock and without rebuilding and hashing the view timeline on every request. Today every file system view request (with the default refresh check on) enters `synchronized (view)` unconditionally and builds a filtered copy of the view timeline (`filterCompletedOrMajorOrMinorCompactionInstants`), whose constructor SHA-256 hashes every instant; the same filter and hash run again after the handler. Between two syncs of the server view that filtered timeline and its hash never change, so the work can be done once per view timeline, and requests from clients that are already in sync do not need the lock. The fix must keep the guarantee from #8079 that a request is never answered from a half-synced view. **Why this task is needed:** Each request costs two O(active instants) hashes, and all requests for one table queue on one monitor while hashing, even when nothing needs to be refreshed. Requests per commit scale with tasks and partitions (index and small-file lookups, base-file lookups in update tasks, clean and clustering planners), so large tables with wide stages serialize on this lock. 64 concurrent up-to-date requests filter and hash the view timeline 128 times, an up-to-date request blocks while another thread holds the view monitor, and a local micro-benchmark (mocked view, 32 client threads, 2000 active instants, on a heavily loaded laptop) tops out around 1400 req/s with p50 17 ms, and around 740 req/s when the timeline ends in a clean (the trailing-clean check hashes the timeline once more under the lock). ### Task Type Performance optimization ### Related Issues **Parent feature issue:** #20257 **Related issues:** #8079 (moved the check under the unconditional lock), #19946 (trailing-clean refresh exemption in the same check), #20239 (open, same method) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
