Doris-Breakwater commented on issue #67619: URL: https://github.com/apache/doris/issues/67619#issuecomment-5570788788
Breakwater-GitHub-Analysis-Slot: slot_b7486a508692 ### Initial maintainer triage **Assessment:** this is a reproducible BE process crash and should be treated as a high-severity memory-safety issue. The supplied version is internally consistent: in the repository, both `4.0.8` and `4.0.8-rc02` point to `bc8ea1bac6d`. The issue currently has no labels, assignee, milestone, or linked development work. The evidence is consistent with a use-after-free, double destruction, or earlier heap corruption, but it is **not yet sufficient to prove one shared MoW or LRU-cache root cause**. The three stacks are destruction sites and may be victims of damage that happened earlier. ### What the 4.0.8 code establishes - In [`LRUCache::insert`](https://github.com/apache/doris/blob/bc8ea1bac6d62d92cdebd2e3ac33cc668961ec36/be/src/olap/lru_cache.cpp#L482-L532), eviction, removal from the hash/LRU structures, and the reference-count transition to zero happen under the shard mutex. Entries exposed through a cache handle have `refs > 1` and are not on the evictable LRU list. Destruction is intentionally deferred until after detachment. Therefore, `free()` being outside `_mutex` is not by itself evidence of a race; crash B still requires an invalid/corrupted handle or value, a double release, or corruption inside the value/destructor path. - Crash A reaches destruction of a `SegmentCache::CacheValue`, then `Segment`, then the `ColumnReaderCache` list and a `ColumnReader` `shared_ptr`. While the cache is alive, its list/map accesses are guarded by `_cache_mutex`. The stack proves that a reader/control structure was invalid at destruction time, but not whether it was damaged by `ColumnReaderCache`, by a stale `Segment` user, or elsewhere. - For crash C, [`InvertedIndexColumnWriter::finish`](https://github.com/apache/doris/blob/bc8ea1bac6d62d92cdebd2e3ac33cc668961ec36/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp#L636-L700) closes and resets the CLucene `IndexWriter`; the segment writer then destroys its column writers before [`IndexFileWriter::finish_close`](https://github.com/apache/doris/blob/bc8ea1bac6d62d92cdebd2e3ac33cc668961ec36/be/src/olap/rowset/segment_v2/index_file_writer.cpp#L233-L254) clears the directory map. [`DorisRAMFSDirectory::~DorisRAMFSDirectory`](https://github.com/apache/doris/blob/bc8ea1bac6d62d92cdebd2e3ac33cc668961ec36/be/src/olap/rowset/segment_v2/inverted_index_fs_directory.cpp#L732-L742) also holds its own mutex while deleting the owned `RAMFile`s. No obvious writer-before-directory ordering violation is visible in this path. A fault attributed to the empty `RAMFile` destructor can occur while its buffer vector is being destroyed, so a core/ASAN report is needed to identify the invalid object. - `SIGSEGV @0x8` is a null-adjacent access, not a unique signature. In particular, the address alone does not prove that a `shared_ptr` control-block field at offset 8 was the source; the faulting instruction and registers are needed. - Serial SQL statements do not make the BE execution serial: one load still uses memtable flush workers, delete-bitmap workers, index writers, and background storage threads. A concurrency/lifetime defect remains possible. - I found no post-4.0.8 change on the local `branch-4.0` history that addresses these LRU, column-reader-cache, or RAM-directory destruction signatures. Therefore, there is not currently an evidence-backed upgrade or existing patch to cite as the fix. Issue #64826 is directionally similar, but the shared small fault address and cache-destruction frame are not enough to declare the same root cause. Disabling a cache or the RAM directory removes a destruction path and also changes allocation layout, pressure, and timing. The mitigation results are useful isolation evidence, but do not establish that each disabled subsystem is faulty or that the remaining configuration is safe. ### Information needed to confirm the root cause 1. Please capture one **complete, non-truncated core** together with the exact `doris_be` binary, build ID, and matching debug symbols. Set the systemd-coredump size limits above the possible BE RSS before the next run. The most useful first artifacts are `thread apply all bt full`, `info registers`, and disassembly around the faulting PC. For A/B, inspect the `LRUHandle` (`value`, `refs`, `in_cache`, `hash`, `next`/`prev`) and the concrete value object; for C, inspect `filesMap`, the failing `RAMFile`, and its `buffers` vector. 2. Provide the full, unredacted crash sections and BE INFO/WARNING logs for several minutes before each crash, including absolute PCs, thread names/IDs, all frames, and any allocator warnings such as double-free/corruption diagnostics. Also include the effective BE configuration and dynamic config changes, not only the static `be.conf`. 3. Identify whether the same target table/data slice fails repeatedly. Please provide the exact failing table DDL (types, declared string lengths, nullability, all index properties), the failing statement/query ID and row count, maximum and representative text lengths, and whether malformed UTF-8 is present. Names and business values can remain redacted. 4. If the environment can be used for a diagnostic build, reproduce under **ASAN** at `bc8ea1bac6d` (or first on a reduced but still reproducing data slice). ASAN is much more likely than a release core to report the original invalid write/free rather than the later destructor victim. Do not combine speculative fixes in that run. ### Suggested next steps 1. Add bug/storage-load ownership and a high-severity/process-crash classification, then assign a BE storage maintainer. 2. Prioritize the full core or ASAN reproduction before changing the three destructors or moving deletion back under the LRU mutex. 3. On a staging node, isolate one variable per run with the same data: (a) MoW with the inverted indexes removed, caches enabled; (b) a non-MoW/Duplicate-Key target with the same inverted indexes; (c) MoW plus the same indexes with only `inverted_index_ram_dir_enable=false`; then compare with the cache-only gates already tested. This separates the MoW delete-bitmap path, inverted-index construction, and allocation/timing effects. 4. The reported configuration switches may be kept as temporary operational mitigations if necessary, but they should not be presented as a root-cause fix until a diagnostic run shows which invariant is violated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
