liangkun9302 opened a new issue, #70496: URL: https://github.com/apache/doris/issues/70496
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version Apache Doris 4.1.5 BE Git Commit: 917b20be838 The issue was also present in Apache Doris 4.1.4. Upgrading from 4.1.4 to 4.1.5 did not resolve the crashes. ### What's Wrong? ## Description We are experiencing **frequent and recurring Apache Doris Backend (BE) process crashes across all three BE nodes in our production cluster**. The issue was already present in **Doris 4.1.4**, and upgrading to **Doris 4.1.5** did not resolve it. We have observed multiple crash signatures: 1. **SIGABRT** caused by a fatal assertion in `brpc::Stream::OnReceived()` (`stream.cpp:470`). 2. **SIGSEGV** in `bthread::TaskGroup::sched_to()` and related bthread functions. 3. **SIGSEGV** during background compaction task selection in `TabletManager::find_best_tablets_to_compaction()`. The crashes occur repeatedly on different BE nodes and are significantly affecting production availability, causing interrupted queries, failed requests, and frequent BE restarts. We are particularly concerned about the repeated BRPC fatal assertion because we have confirmed the same stack trace on multiple BE nodes. ## Environment | Item | Value | |---|---| | Doris version | 4.1.5 | | BE Git commit | `917b20be838` | | Previous version | 4.1.4 (same general instability was observed) | | Cluster topology | 3 FE + 3 BE | | Operating system | Enterprise Linux 9 compatible | | Linux kernel | `5.14.0-570.58.1.el9_6.x86_64` | | glibc | 2.34 | | Java | OpenJDK 17 | | BE executable SHA256 | `acd3397d0b41d7449b343bae72dc04fcdbc2d2c525cf263923e1f9cbdeb08a0e` | All three BE nodes have the same BE executable SHA256, Linux kernel version, and glibc version. Each BE has approximately 125 GB of memory and 112 reported CPU cores. ## Crash Signature 1: BRPC Fatal Assertion The following error has occurred repeatedly across multiple BE nodes: ```text F20261112 11:04:39.427426 425802 stream.cpp:470] Check failed: false Fail to push into channel *** Check failure stack trace: *** google::LogMessageFatal::~LogMessageFatal() brpc::Stream::OnReceived() brpc::policy::ParseStreamingMessage() brpc::InputMessenger::CutInputMessage() brpc::InputMessenger::ProcessNewMessage() brpc::InputMessenger::OnNewMessages() brpc::Socket::ProcessEvent() bthread::TaskGroup::task_runner() bthread_make_fcontext *** Query id: 0-0 *** *** tablet id: 0 *** *** Current BE git commitID: 917b20be838 *** *** SIGABRT received by PID 422506 *** ``` An identical function-level stack trace was observed on another BE node on November 11, 2026: ```text F20261111 14:28:02.149678 1981169 stream.cpp:470] Check failed: false Fail to push into channel *** Check failure stack trace: *** google::LogMessageFatal::~LogMessageFatal() brpc::Stream::OnReceived() brpc::policy::ParseStreamingMessage() brpc::InputMessenger::CutInputMessage() brpc::InputMessenger::ProcessNewMessage() brpc::InputMessenger::OnNewMessages() brpc::Socket::ProcessEvent() bthread::TaskGroup::task_runner() bthread_make_fcontext *** Query id: 0-0 *** *** tablet id: 0 *** *** Current BE git commitID: 917b20be838 *** *** SIGABRT received by PID 1977966 *** ``` Other fatal assertions have also been observed: ```text stream.cpp:113 Check failed: 0 == Socket::Address(fake_sock_id, &ptr) (0 vs. -1) ``` These failures have occurred on all three BE nodes. ## Crash Signature 2: SIGSEGV in bthread We also observed recurring segmentation faults involving bthread scheduling. Example: ```text SIGSEGV address not mapped to object (@0x38) bthread::TaskGroup::sched_to() bthread::TaskGroup::run_main_task() bthread::TaskControl::worker_thread() ``` Similar faults have occurred at addresses including: ```text 0x0 0x8 0x18 0x20 0x38 0x50 ``` We have not yet confirmed whether these SIGSEGV crashes share the same underlying root cause as the BRPC fatal assertions. ## Crash Signature 3: SIGSEGV During Compaction Scheduling Another BE crash occurred during background compaction task generation: ```text SIGSEGV address not mapped to object (@0x200) TabletManager::find_best_tablets_to_compaction() TabletManager::for_each_tablet() StorageEngine::_generate_compaction_tasks() StorageEngine::_compaction_tasks_producer_callback() ``` The relevant crash occurred on November 11, 2026, at approximately 14:29:34 CST (UTC+8). ## Examples of Crash Timeline All timestamps below are CST (UTC+8). | Time | BE node | Failure | |---|---|---| | 2026-11-11 14:28:02 | BE-2 | BRPC `stream.cpp:470` SIGABRT | | 2026-11-11 14:28:24 | BE-3 | SIGSEGV | | 2026-11-11 14:29:34 | BE-3 | Compaction-related SIGSEGV | | 2026-11-11 18:50:28 | BE-3 | SIGSEGV | | 2026-11-11 18:51:19 | BE-1 | SIGSEGV | | 2026-11-11 18:53:38 | BE-2 | BRPC `stream.cpp:470` SIGABRT | | 2026-11-11 18:54:18 | BE-3 | SIGSEGV | | 2026-11-12 11:04:39 | BE-1 | BRPC `stream.cpp:470` SIGABRT | Some crashes occurred very close together across different BE nodes, suggesting a possible correlation, although we have not established a causal relationship. ## Workload and Troubleshooting Performed Our workload includes analytical SELECT queries as well as INSERT, UPDATE, and DELETE operations. We investigated whether high-frequency small writes or compaction overload might be responsible. Findings: - Typical cluster FE QPS is approximately 15–25. - Cumulative and Base Compaction Scores show occasional spikes but no clear sustained backlog in the inspected monitoring windows. - Compaction permit waiting was zero on all three BEs at the time of inspection. - Compaction throughput was relatively low at the time of inspection. - The configured `max_tablet_version_num` is 3000 on all BEs. - A sampled table had a maximum `VersionCount` of only 5. - At the time of the November 11, 14:28–14:29 crashes, SQL audit logs recorded 56 INSERTs, 2 UPDATEs, and 0 DELETEs in the corresponding five-minute interval. These observations do not indicate obvious sustained compaction overload or unusually high SQL request volume during the initial crash window. We have not identified a specific SQL statement that reliably reproduces the issue. We have not disabled automatic compaction as a production workaround because other crashes also involve BRPC and bthread. ## Core Dumps Systemd-coredump recorded repeated SIGSEGV and SIGABRT events for `/data/doris/be/lib/doris_be` across all three BE nodes. Most collected core dumps are marked as `truncated`, so the BE log stack traces are currently our most reliable evidence. We can provide additional sanitized logs, configuration information, and diagnostic outputs if needed. ## How to Reproduce We do not yet have a minimal deterministic reproduction. The issue occurs during normal production operation with a three-BE cluster. The BE processes may crash during distributed processing, BRPC streaming message handling, or background tasks. BE processes are restarted by our watchdog, but repeated crashes continue. The issue was observed on Doris 4.1.4 and remains present on 4.1.5. ## Expected Behavior - Normal distributed query execution and BE-to-BE communication should not terminate a BE process. - Recoverable BRPC stream or execution queue errors should be handled safely without causing the entire BE process to abort. - Background compaction and bthread scheduling should not cause segmentation faults. - Temporary network or stream failures should be handled without repeated cluster-wide instability. ## Actual Behavior BE processes repeatedly terminate due to SIGABRT or SIGSEGV. The BRPC fatal assertion is consistently reproducible at the stack-signature level across multiple BE nodes, although the exact triggering workload is not yet known. Frequent BE restarts are disrupting production availability. ## Questions / Request for Help This is a production-impacting stability issue. We would appreciate help from the Doris maintainers with the following: 1. Is the `brpc::Stream::OnReceived()` failure at `stream.cpp:470` a known issue in Doris 4.1.4 or 4.1.5? 2. Could this be related to BRPC Stream / ExecutionQueue lifecycle management or a concurrency race? 3. Are the `stream.cpp:113`, bthread SIGSEGV, and compaction SIGSEGV signatures potentially related, or should they be investigated separately? 4. Is there an existing fix, backport, stable release, or tested patch that addresses the BRPC fatal assertion? 5. Is there a **safe temporary workaround** to reduce BE crashes without disabling essential storage maintenance functionality? 6. What additional diagnostics would be most useful to identify the root cause? **We would particularly appreciate an actionable workaround or patch recommendation, as the issue is currently affecting production services.** Thank you for your help. ### What You Expected? We expect all Doris BE processes to remain stable during normal production workloads. BRPC streaming, bthread scheduling, and background compaction should not cause BE processes to crash with SIGABRT or SIGSEGV. If an internal stream or execution queue becomes unavailable, the error should be handled gracefully without terminating the entire BE process. Queries and data ingestion should continue without frequent BE restarts. ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
