stinger1206 commented on issue #70496: URL: https://github.com/apache/doris/issues/70496#issuecomment-7033219298
@liangkun9302 @leeks8888 @yiguolei — sharing some findings from our investigation that might be relevant to your crashes. We've been investigating similar unexplained BE crashes in our production Doris cluster for several weeks and have some findings that might be relevant to your case. **Our environment:** - Apache Doris 4.1.5, coupled mode, 6 BEs on Kubernetes. - AlmaLinux 9.8. - Kernel before the upgrade: `5.14.0-687.5.4.el9_8`. - Kernel after the upgrade: `5.14.0-687.64.1.el9_8`. - jemalloc with Doris's default `muzzy_decay_ms:5000` configuration (`MADV_FREE`). - THP disabled, automatic NUMA balancing enabled. We observed crashes in different BE components, including brpc/bthread and bvar. We initially suspected a concurrency or object-lifetime issue in brpc. We spent several weeks analyzing core dumps, inspecting the relevant source code, building instrumented binaries, and running dedicated concurrency stress tests with ASan. Despite billions of operations and extensive forced bthread migration testing, we could not reproduce the suspected corruption through the migration path. **What eventually made a difference was upgrading the host kernel.** Between November 3 and November 6, we recorded 46 BE crashes. We performed a rolling kernel upgrade across our Kubernetes workers on November 5–6. Interestingly, each worker stopped experiencing these crashes after its individual kernel upgrade and reboot, while workers still running the old kernel continued crashing. The Doris BE binary remained unchanged. Following the upgrade, we observed no further crashes of this kind for more than 14 days, even after increasing the production workload by approximately 3–5 times. **Possible kernel memory corruption** While investigating the kernel changelogs, we identified several memory-management fixes that could potentially explain this behavior. One particularly interesting candidate is: `x86/mm: Add missing saved dirty bit to page protection change mask` (RHEL-220787) This fix appears in the EL9.8 kernel changelog starting with `5.14.0-687.39.1.el9_8`. There has also been a recent investigation in the DuckDB community involving Linux kernel memory corruption associated with jemalloc and `MADV_FREE`: https://github.com/duckdb/duckdb/pull/28267 Related kernel vulnerability: https://ubuntu.com/security/CVE-2026-97945 The latter specifically involves transparent huge pages, so it is unlikely to be the exact mechanism in our environment, where THP is disabled. However, it demonstrates that allocator/kernel interactions can cause unexpected user-space memory corruption and misleading crash signatures in database engines. We haven't isolated the exact kernel patch responsible for resolving our crashes, so we cannot claim a definitive root cause yet. Nevertheless, the per-node correlation with the rolling kernel upgrade is quite compelling. Since your environment also uses an Enterprise Linux 9 kernel, I would strongly recommend testing a newer kernel on one of your BE hosts while keeping the Doris binary and configuration unchanged. It would be interesting to see whether a kernel upgrade has a similar effect in your environment. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
