matanper opened a new issue, #68972:
URL: https://github.com/apache/doris/issues/68972

   ### Search before asking
   
   - [x] I had searched in the 
[issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no 
similar issues.
   
   ### Version
   
   4.1.4 (cloud mode, arm64). master is the same:
   - `be/src/io/cache/cached_remote_file_reader.cpp:790-796`: on a failed read, 
the direct-read loop just `break`s, with the comment `// TODO: maybe read 
failed because block evict, should handle error`.
   - `be/src/io/cache/cached_remote_file_reader.cpp:178-185`: 
`_insert_file_reader` uses `_cache_file_readers.emplace(...)`, so a block that 
is downloaded again cannot replace the stale entry.
   - `be/src/io/cache/block_file_cache.h:164-166`: a block whose only extra 
reference is the reader's direct-read map (`_owned_by_cached_reader`) is 
`releasable()`, so it can be evicted.
   
   ### Description
   
   With `enable_read_cache_file_directly` (default true), 
`CachedRemoteFileReader` keeps a map from offset to `FileBlockSPtr` 
(`_cache_file_readers`), and reads those blocks without going through the cache 
lock. These blocks are evictable on purpose. So eviction can remove a block and 
delete its file while a reader still has the block in its map.
   
   The entry is never dropped:
   
   1. The next direct read of that range calls `FileBlock::read`. That fails to 
open the deleted file, logs `open file failed` in `FSFileCacheStorage::read` 
(`fs_file_cache_storage.cpp:384/397`), and the loop breaks. The read then falls 
back to the indirect path.
   2. The indirect path downloads the block again and calls 
`_insert_file_reader` with the new block. `emplace` does not overwrite the 
existing key, so the stale block stays.
   3. Every later read of that offset by the same reader repeats steps 1 and 2: 
a failed open, a warning, and a fallback.
   
   Results are still correct, because the reader falls back. But each read of 
the range pays a failed `open()` plus the indirect path. On busy BEs this shows 
up as bursts of `open file failed` warnings for the same files, which is how we 
found it. The indirect path already heals a stale `DOWNLOADED` entry on 
`NOT_FOUND` (`need_self_heal`, `cached_remote_file_reader.cpp:1031-1066`), but 
the direct path does not.
   
   ### Solution
   
   - When a direct read fails, drop the entry from `_cache_file_readers`. Do it 
after releasing the shared lock, and only if the entry still maps the same 
block, then fall back as today.
   - In `_insert_file_reader`, use `insert_or_assign`, so a block downloaded 
again replaces the stale one.
   - Add a bvar that counts dropped stale blocks.
   
   ### How to Reproduce?
   
   1. Keep `enable_read_cache_file_directly` on (the default) and use a small 
file cache.
   2. Open a reader on a segment and read a range, so the block is cached and 
recorded in the reader's direct-read map.
   3. Evict or clear that block from the cache (for example, cache pressure 
from other data, or a file cache clear).
   4. Read the same range again with the same reader, several times. Each read 
logs `open file failed` and goes through the indirect path.
   
   We have a unit test, `direct_read_drops_evicted_block_and_maps_replacement`, 
that fails on the current code.
   
   ### Anything Else?
   
   Not covered here: stale metadata after a restart. Cache loaders trust the 
stored block metadata without checking that the file exists. Readers recover 
from that through the remote fallback. Related but separate: #68348 (blocks 
marked deleting while busy are never reclaimed; `_cache_file_readers` is one of 
the holders it lists).
   
   ### Are you willing to submit PR?
   
   - [x] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to