Doris-Breakwater commented on issue #68972:
URL: https://github.com/apache/doris/issues/68972#issuecomment-6123641768

   The stale direct-read map entry is supported by code inspection and warrants 
a BE file-cache fix. The stronger claim that *every* subsequent read must fail 
after a successful re-download needs qualification. This is static analysis, 
not a runtime reproduction; I have not run the reported unit test.
   
   I inspected the local `4.1.4` tag 
(`ad35a140c7fd0b842f18c23300bac581f7d04326`) and upstream master snapshot 
`cad4abb82d40336178a96b3d2069aed03a6ef38c`. Both retain the reported `emplace` 
and failure-without-invalidation behavior. The issue is currently open with no 
labels; the relevant area is BE / file cache / direct-read lifecycle.
   
   **Verified mechanism**
   
   - `_can_read_cache_file_directly()` gates this path on Doris-table readers 
and `enable_read_cache_file_directly`. `_insert_file_reader()` sets 
`_owned_by_cached_reader` and uses `emplace`, leaving an existing offset mapped 
to the old object. [Reader initialization and 
insertion](https://github.com/apache/doris/blob/cad4abb82d40336178a96b3d2069aed03a6ef38c/be/src/io/cache/cached_remote_file_reader.cpp#L130-L190).
   - `FileBlockCell::releasable()` explicitly permits eviction when the 
references are the cache cell plus a direct-reader reference. Eviction removes 
the cache cell and eventually the storage file; it does not erase the reader's 
map. [Eviction 
eligibility](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/be/src/io/cache/block_file_cache.h#L159-L166).
   - The direct loop holds the reader's shared mutex, calls `FileBlock::read`, 
and breaks on failure without invalidating its entry. Successfully read prefix 
bytes are retained; the remaining range goes through the fallback path. In the 
synchronous cache-write path, successful download/finalization attempts 
insertion again, but the existing key wins. Indirect `NOT_FOUND` self-healing 
clears cache metadata, not this reader-local map. [Direct-read 
loop](https://github.com/apache/doris/blob/cad4abb82d40336178a96b3d2069aed03a6ef38c/be/src/io/cache/cached_remote_file_reader.cpp#L758-L817).
   
   **Qualification and impact**
   
   `FileBlock::read` delegates to storage using its key. Storage resolves the 
file descriptor and filename by **hash and offset**, rather than by `FileBlock` 
object identity. Consequently, once a replacement file at that key is finalized 
and remains present, the old object can successfully read it. Repeated failed 
opens require the file to remain missing or become missing again—for example, 
further eviction, an unfinished/failed cache write, or cache admission being 
skipped. These are possibilities to check, not established causes of the 
reported warning bursts. [Storage lookup and 
deletion](https://github.com/apache/doris/blob/cad4abb82d40336178a96b3d2069aed03a6ef38c/be/src/io/cache/fs_file_cache_storage.cpp#L365-L414).
   
   There is also a concrete consequence even when such a read succeeds: the 
direct path queues the old object for an LRU update, while `update_block_lru()` 
rejects objects whose identity differs from the current cache cell. Those reads 
therefore do not refresh the replacement block's LRU through that path. [LRU 
identity 
check](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/be/src/io/cache/block_file_cache.cpp#L631-L654).
 Warning/fallback overhead is plausible; its frequency and latency impact are 
unquantified. The inspected fallback preserves read correctness when its source 
read succeeds; no evidence here establishes data corruption.
   
   **Suggested PR and validation**
   
   1. Retain the failed offset and a `FileBlockSPtr` under the shared lock, 
release that lock, then acquire exclusive `_mtx` and erase only if the current 
entry still points to the failed object. Do not retain an iterator across the 
unlock. Preserve prefix-byte accounting and the existing fallback.
   2. Use replacement-capable insertion, such as `insert_or_assign`, under 
`_mtx`. Review `_owned_by_cached_reader` bookkeeping on insertion, replacement, 
erasure, and destruction, including multiple readers; the shared boolean should 
not be blindly cleared while another reader still holds the block.
   3. Count actual map invalidations. If all read errors trigger invalidation, 
name the counter accordingly or distinguish `NOT_FOUND` from other errors; a 
failed read alone does not prove eviction.
   4. Extend the proposed regression test to assert object replacement, correct 
returned bytes, and recovery to direct reads after completed re-download 
without further eviction. Cover a successful prefix followed by a missing 
block, and a concurrent replacement between failure and exclusive-lock cleanup. 
Synchronize actual file deletion and release extra test-held references so 
eviction eligibility is exercised.
   
   Please attach the named test and its failing assertion/output, the deployed 
BE build SHA, sanitized full warning statuses with timestamps for one repeated 
hash/offset, cache capacity/clear activity, and effective cache-write mode 
(`enable_async_file_cache_write` plus any override). State whether replacement 
finalization completed before the next read and whether another eviction 
occurred. This distinguishes the proven stale-object defect from persistent 
missing-file behavior. Keep restart metadata recovery and the 
deleting-while-busy lifecycle in #68348 separate.
   
   Breakwater-GitHub-Analysis-Slot: slot_d95a0e1d3269
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to