matanper opened a new issue, #68893: URL: https://github.com/apache/doris/issues/68893
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version 4.1.4 (cloud mode); the reactive eviction path is the same on master. ### What's Wrong? In `BlockFileCache`, only the background "evict in advance" path removes block files asynchronously (via `_recycle_keys`). On the reactive path, a `try_reserve*` that must evict right now to make room uses `sync = !evict_in_advance`, i.e. synchronous removal. That makes `remove_file_blocks` call `_storage->remove(key)`, a filesystem unlink, while still holding the global `cache_lock`. Once the cache is full, almost every reservation takes the reactive path. Filesystem I/O then runs inside the global lock, and all cache readers and writers queue behind the disk. Throughput declines steadily as the cache fills, and recovers if the cache is emptied. ### What You Expected? No filesystem I/O while holding the global cache lock: unlinks deferred to after the lock is released, or to the existing recycle path, on the reactive path too. ### How to Reproduce? Run sustained ingestion with write-through to the file cache until it is full, then compare throughput and lock wait times before and after the cache fills. Off-CPU stacks show threads in `unlink` under `remove_file_blocks` while others wait on `cache_lock`. ### Anything Else? Collecting the keys to remove under the lock and doing the storage removal after releasing it removed the decline in our tests. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
