Doris-Breakwater commented on issue #68972: URL: https://github.com/apache/doris/issues/68972#issuecomment-6123641768
The stale direct-read map entry is supported by code inspection and warrants a BE file-cache fix. The stronger claim that *every* subsequent read must fail after a successful re-download needs qualification. This is static analysis, not a runtime reproduction; I have not run the reported unit test. I inspected the local `4.1.4` tag (`ad35a140c7fd0b842f18c23300bac581f7d04326`) and upstream master snapshot `cad4abb82d40336178a96b3d2069aed03a6ef38c`. Both retain the reported `emplace` and failure-without-invalidation behavior. The issue is currently open with no labels; the relevant area is BE / file cache / direct-read lifecycle. **Verified mechanism** - `_can_read_cache_file_directly()` gates this path on Doris-table readers and `enable_read_cache_file_directly`. `_insert_file_reader()` sets `_owned_by_cached_reader` and uses `emplace`, leaving an existing offset mapped to the old object. [Reader initialization and insertion](https://github.com/apache/doris/blob/cad4abb82d40336178a96b3d2069aed03a6ef38c/be/src/io/cache/cached_remote_file_reader.cpp#L130-L190). - `FileBlockCell::releasable()` explicitly permits eviction when the references are the cache cell plus a direct-reader reference. Eviction removes the cache cell and eventually the storage file; it does not erase the reader's map. [Eviction eligibility](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/be/src/io/cache/block_file_cache.h#L159-L166). - The direct loop holds the reader's shared mutex, calls `FileBlock::read`, and breaks on failure without invalidating its entry. Successfully read prefix bytes are retained; the remaining range goes through the fallback path. In the synchronous cache-write path, successful download/finalization attempts insertion again, but the existing key wins. Indirect `NOT_FOUND` self-healing clears cache metadata, not this reader-local map. [Direct-read loop](https://github.com/apache/doris/blob/cad4abb82d40336178a96b3d2069aed03a6ef38c/be/src/io/cache/cached_remote_file_reader.cpp#L758-L817). **Qualification and impact** `FileBlock::read` delegates to storage using its key. Storage resolves the file descriptor and filename by **hash and offset**, rather than by `FileBlock` object identity. Consequently, once a replacement file at that key is finalized and remains present, the old object can successfully read it. Repeated failed opens require the file to remain missing or become missing again—for example, further eviction, an unfinished/failed cache write, or cache admission being skipped. These are possibilities to check, not established causes of the reported warning bursts. [Storage lookup and deletion](https://github.com/apache/doris/blob/cad4abb82d40336178a96b3d2069aed03a6ef38c/be/src/io/cache/fs_file_cache_storage.cpp#L365-L414). There is also a concrete consequence even when such a read succeeds: the direct path queues the old object for an LRU update, while `update_block_lru()` rejects objects whose identity differs from the current cache cell. Those reads therefore do not refresh the replacement block's LRU through that path. [LRU identity check](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/be/src/io/cache/block_file_cache.cpp#L631-L654). Warning/fallback overhead is plausible; its frequency and latency impact are unquantified. The inspected fallback preserves read correctness when its source read succeeds; no evidence here establishes data corruption. **Suggested PR and validation** 1. Retain the failed offset and a `FileBlockSPtr` under the shared lock, release that lock, then acquire exclusive `_mtx` and erase only if the current entry still points to the failed object. Do not retain an iterator across the unlock. Preserve prefix-byte accounting and the existing fallback. 2. Use replacement-capable insertion, such as `insert_or_assign`, under `_mtx`. Review `_owned_by_cached_reader` bookkeeping on insertion, replacement, erasure, and destruction, including multiple readers; the shared boolean should not be blindly cleared while another reader still holds the block. 3. Count actual map invalidations. If all read errors trigger invalidation, name the counter accordingly or distinguish `NOT_FOUND` from other errors; a failed read alone does not prove eviction. 4. Extend the proposed regression test to assert object replacement, correct returned bytes, and recovery to direct reads after completed re-download without further eviction. Cover a successful prefix followed by a missing block, and a concurrent replacement between failure and exclusive-lock cleanup. Synchronize actual file deletion and release extra test-held references so eviction eligibility is exercised. Please attach the named test and its failing assertion/output, the deployed BE build SHA, sanitized full warning statuses with timestamps for one repeated hash/offset, cache capacity/clear activity, and effective cache-write mode (`enable_async_file_cache_write` plus any override). State whether replacement finalization completed before the next read and whether another eviction occurred. This distinguishes the proven stale-object defect from persistent missing-file behavior. Keep restart metadata recovery and the deleting-while-busy lifecycle in #68348 separate. Breakwater-GitHub-Analysis-Slot: slot_d95a0e1d3269 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
