Doris-Breakwater commented on issue #66895: URL: https://github.com/apache/doris/issues/66895#issuecomment-5326497830
Breakwater-GitHub-Analysis-Slot: slot_801abf44ffa5 ### Initial assessment This is a credible, high-impact BE stability report. The missing `apache/brpc#3291` follow-up is a strong candidate for an incomplete `#64040` backport, but the current crash stack is not sufficient to declare it the confirmed root cause. ### Verified facts - Doris 4.1.3 still builds brpc 1.4.0 and applies [`brpc-1.4.0-fix-agent-combiner-thread-safety.patch`](https://github.com/apache/doris/blob/4.1.3/thirdparty/patches/brpc-1.4.0-fix-agent-combiner-thread-safety.patch). That patch contains the shared/weak-pointer lifetime changes from `apache/brpc#2949` (plus its constructor follow-up), as intended by `#64040`. - The 4.1.3 patch does **not** change the inherited `AgentCombiner` destructor, so the patched code still executes `~AgentCombiner() -> clear_all_agents()` and walks `_agents` before destroying the agent ID. - [`apache/brpc#3291`](https://github.com/apache/brpc/pull/3291) demonstrates, under ASAN, a second heap-use-after-free introduced/exposed by the weak-pointer design: once the final `shared_ptr` is released, an `Agent`'s `weak_ptr` is already expired; concurrent TLS teardown can therefore free the `Agent` without unlinking it while `clear_all_agents()` is traversing the same list. The upstream fix deliberately stops traversing `_agents` from `~AgentCombiner()`. - The current Doris `master` and `branch-4.1` versions of this brpc patch are still the same patch blob as 4.1.3 and do not contain the `#3291` follow-up. - The reported memory headroom does not rule this out: a lifetime UAF is independent of ordinary RSS/process-limit pressure. ### What remains unverified - Upstream `#3291`'s direct ASAN failure is in `~AgentCombiner()/clear_all_agents()`, whereas the reported immediate frame is the sampling thread in `SeriesSampler::take_sample()`. `take_sample()` calls `Reducer::get_value()`, which in turn walks the combiner's agent list, so the frame is compatible with an invalid/corrupted agent list; however, it does not by itself identify which lifetime race corrupted the list. The exact PC/file:line and core state are required to connect this crash conclusively to `#3291`. - There is not yet evidence that the V2-to-V3 rewrite path itself is defective. It may only provide the concurrency/thread churn that makes a bvar lifetime race more likely. ### Information needed from the reporter 1. The exact BE build commit/build ID (the startup build-version lines are sufficient), not only the `4.1.3` release string. 2. For both crashes, the full symbolized crashing-thread trace with exact PC and file:line. Ideally provide the cores plus the matching unstripped `doris_be` and debug symbols. Useful GDB output includes `thread apply all bt full`, `info registers`, `info line *$pc`, and instructions around `$pc`. Please use a private maintainer-approved channel if a core may contain sensitive data. 3. Whether both crashes have the same instruction offset inside `SeriesSampler::take_sample()`, and the fault address from each core. 4. BE logs covering several minutes before each crash, plus the exact table-rewrite command/job type and its timing on each affected BE. Any other concurrent schema-change, compaction, load, or BE restart/thread-shutdown activity would also be useful. 5. If feasible, an ASAN run of the upstream `MultiDimensionTest.shared` reproducer against Doris's patched brpc, followed by the same run with the `#3291` change. ### Recommended maintainer actions - Add `area/brpc` and `kind/stability` (the issue currently has no labels). - Prepare and validate a backport of `apache/brpc#3291` into the Doris brpc patch for `master` and `branch-4.1`. The upstream ASAN reproducer makes this follow-up independently justified even before it is proven to explain these two production cores. - Validate with ASAN and a thread-create/exit stress test while bvar reducers are exposed and sampled, then perform a sustained workload soak. - Use the core to verify whether the failing instruction is in `AgentCombiner::combine_agents()`/linked-list traversal. If it is not, continue with the sampler-owner lifetime and broader heap-corruption investigation rather than treating the backport as a complete diagnosis. No code change was made during this triage. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
