leeks8888 commented on issue #70496:
URL: https://github.com/apache/doris/issues/70496#issuecomment-6904163366
We see the same crash families on **Apache Doris 3.1.4** (BE commit
`7f5ba43de6`, official `apache-doris-3.1.4-bin-x64-noavx2` binary, brpc 1.4.0),
so this does not look specific to 4.1.x. Sharing our data and a core-dump
analysis in case it helps narrow it down.
### Environment
| Item | Value |
|---|---|
| Doris | 3.1.4 (`7f5ba43de6`), 3 FE + 28 BE |
| BE host | Rocky Linux 9, kernel `5.14.0-611.5.1.el9_7` |
| BE runtime | Docker, Ubuntu 22.04 base (glibc 2.35), OpenJDK 17.0.21 |
| BE size | 16 cores (one node 32), 125 GB RAM, `mem_limit = 80%` |
The crashes started before we moved the image from Ubuntu 20.04 to 22.04 and
have the same rate after, so the base image / glibc is not the trigger.
### Crash statistics (BE `be.out`, 2026-05-01 – 2026-11-13, 991 crashes)
| Family | Count | Top frames |
|---|---|---|
| bthread SIGSEGV | 663 (67%) | `bthread::TaskControl::worker_thread`,
`TaskGroup::task_runner` / `usleep`, `ExecutionQueueBase::_execute_tasks`;
fault addr `0x0` / `0x38` |
| brpc Stream SIGABRT | 109 (11%) | `Stream::Create ← StreamAccept ←
PInternalService::open_load_stream` with `Check failed: 0 ==
Socket::Address(fake_sock_id, &ptr)`; `Stream::OnReceived ←
ParseStreamingMessage` |
| JVM (in-process) | 101 (10%) | C1 `GraphBuilder`,
`SymbolTable::do_lookup`, `Symbol::increment_refcount`,
`ProtectionDomainEntry::object_no_keepalive` |
| other | 118 | `je_large_prof_info_get`,
`VersionGraph::capture_consistent_versions`, `MetadataAdder<TabletMeta>`,
`InvertedIndexFileInfo`, `bvar::Reducer<…>::SeriesSampler::take_sample` (5) |
- Crashes are spread evenly across all 28 BEs (21–47 each), so not a single
bad host.
- Rate follows load: ~7.5–12.6/day on weekdays, ~4/day on weekends, peak
08:00–17:00. Monthly totals grew from 68 (May) to 291 (Oct).
- Ruled out: kernel OOM (none), `vm.max_map_count` (2,000,000), fd limit
(≈5k of 65k), swap, THP.
- Setting `enable_memtable_on_sink_node=false` for ~2 h did not change the
rate. That change only skips the `open_load_stream` path.
- The query id printed in the crash header usually belongs to a SELECT that
had already finished OK, so there is no single culprit query.
The JVM crashes inside the BE process (symbol table / metadata corruption)
and the variety of unrelated victims make us think this is **native heap
corruption (use-after-free or stray write) inside `doris_be`**, with bthread /
brpc / JVM structures as the victims.
### Core dump analysis: bthread SIGSEGV @0x38
We captured a full core (124 GB virtual, 41 GB on disk) of a
`bthread::TaskControl::worker_thread` SIGSEGV. The unwinder stops at the signal
handler, so we read the saved `ucontext` registers directly:
```
RIP = <doris_be>+0xd03b710 ; symbol-less frame, the same "...710" address
in every
; worker_thread SIGSEGV we have
CR2 = 0x38, TRAPNO = 14, ERR = 4, RAX = 0
```
The instruction at RIP is a GCC isolated-NULL-path trap
(`-fisolate-erroneous-paths-dereference`):
```
mov 0x38,%rax
ud2
```
Only `bthread::TaskGroup::run_main_task` branches to that trap, after
`wait_task()` returns a tid. This is the inlined `address_meta(tid)` →
`ResourcePool<TaskMeta>::address_resource()` lookup:
```
rmt+158: call TaskGroup::wait_task
rmt+190: shr $0x18 ; group = _block_groups[id >> 24] ; NULL -> trap
(rmt+205)
rmt+218: block = group->blocks[(id >> 8) & 0xffff] ; NULL -> trap
(rmt+225)
rmt+234: cmp %rax, 0x8000(%r12) ; offset vs block->nitem
rmt+242: jbe trap ; nitem <= offset ->
trap <== taken
```
State in the core:
- tid from the run queue: `0x00000ae2_0005f300` (version `0xae2`, slot
`0x5f300` → block `0x5f3`, item 0)
- the group and block pointers are valid
- **`block->nitem` (block+0x8000) is 0**
- yet **item 0 of that block is a fully constructed `TaskMeta` whose `tid`
field is `0x00000ae20005f300`**, with a valid fn pointer and arg
`get_resource()` increments `nitem` right after constructing the item,
before the tid can be published to any queue. The core is taken after all
threads stopped, so this is not a visibility race. **Something overwrote
`block->nitem` (8 bytes at block+0x8000) with 0 after the TaskMeta was
created.** That is a stray write or use-after-free into memory that the
ResourcePool block now owns.
### bvar AgentCombiner fixes
#64040 / #67276 (brpc#2949 / brpc#3291) fix a use-after-free that writes
zeros into freed TLS agent memory. That pattern would fit the zeroed `nitem`
above. But this issue reports the same crash families on 4.1.5 (`917b20be838` =
4.1.5-rc03, which has both patch files). So either those patches don't cover
the remaining writer, or the release binary was built without rebuilding
thirdparty. Could a maintainer confirm whether the official 4.1.5 binary
actually includes the patched brpc?
We can run gdb commands on the core if that helps; we can't share the core
itself because it contains customer data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]