liangkun9302 commented on issue #70496: URL: https://github.com/apache/doris/issues/70496#issuecomment-6936480178
Thank you @yiguolei, @Doris-Breakwater, @leeks8888, and @stinger1206 for your detailed analysis and suggestions. I really appreciate your help with this issue. Our production cluster is running Doris 4.1.5 (BE commit `917b20be838`), with 3 FE and 3 BE nodes. We previously experienced the same instability on 4.1.4, and upgrading to 4.1.5 did not resolve it. The recurring BE crashes are significantly affecting production availability, so we would like to identify both a safe temporary workaround and the underlying root cause. ### 1. Regarding the suggested configuration changes Thank you @yiguolei for suggesting: - Setting `muzzy_decay_ms:0` in `JEMALLOC_CONF` in `be.conf`. - Setting `enable_memtable_on_sink_node=false`. We will evaluate these changes and, where applicable, test them in a controlled manner, since this is a production environment. Before proceeding, could you please clarify: - Is `muzzy_decay_ms:0` intended to mitigate a known jemalloc-related issue, or is it primarily a diagnostic workaround for possible memory corruption? - @leeks8888 mentioned that disabling `enable_memtable_on_sink_node` did not significantly reduce crashes in their environment. Would you still recommend this as a priority workaround for our case? - Are there any known side effects or performance impacts we should consider when applying these settings in production? ### 2. Possible shared memory corruption The ResourcePool `TaskMeta` corruption reported by @leeks8888 is particularly interesting. Our crashes occur in several different components, including: - `brpc::Stream::OnReceived()` — SIGABRT - `bthread::TaskGroup::sched_to()` — SIGSEGV - `TabletManager::find_best_tablets_to_compaction()` — SIGSEGV Although we cannot yet confirm that these crashes share the same root cause, the similar crash patterns across Doris 3.1.4 and 4.1.5 suggest that investigating possible memory corruption or object lifetime issues may be worthwhile. Could the maintainers help determine whether these failures might be related to a common underlying issue, rather than independent bugs? Also, could you confirm whether the official Doris 4.1.5 binary includes the relevant BRPC fixes, particularly `brpc#2949` and `brpc#3291`? ### 3. Additional diagnostics Most of our existing core dumps were marked as `truncated`. We will investigate the core dump configuration to improve the collection of complete crash data. Could you recommend the most useful additional diagnostics for identifying the root cause, especially for the BRPC and bthread crashes? We can provide sanitized logs, configuration details, and additional diagnostic information if needed. ### 4. Production workaround or fix Since all three BE nodes have experienced repeated crashes, this is a serious production stability issue for us. We would particularly appreciate guidance on whether: - There is an existing patch or newer build that addresses these crash signatures. - There is a recommended and tested workaround for production environments. - The issue should be investigated further by the Doris BE/BRPC maintainers. We are happy to cooperate with further troubleshooting and provide additional evidence. Thank you again for your time and support. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
