felix199103 commented on issue #66997: URL: https://github.com/apache/doris/issues/66997#issuecomment-5364515190
## Follow-up evidence (userspace bt + metrics) Thanks for the earlier guidance. We now have a full userspace backtrace after restarting one BE (thread count low enough for gdb). ### Version - Doris BE `3.1.4-rc02` / commit `7f5ba43de6c7f148e1dacdb7cb838394d2ec1537` - Image based on `apache/doris:be-3.1.4` (+ paimon-s3 jar only) - Symbols path: `/home/zcp/repo_center/doris_release/doris/` ### Workload group Only one WG (`SHOW WORKLOAD GROUPS`): - Id=1 Name=normal - scan_thread_num / max_remote_scan_thread_num / min_remote_scan_thread_num = **-1** (defaults) - BE effective: scan=48, max_remote=512, min_remote=8 ### Metrics right after sampling (same window as bt) Pool id: `a44b3294c7572642-ac5de49ef6b4e883` (`RScan_normal` / workload `normal`) | metric | value | |---|---| | OS threads named RScan_normal | ~235 | | active_threads | 0 | | max_threads | 512 | | queue_size | 0 | | task_execution_count_total | 260 | | submit_failed | 0 | (Previously, before restart, OS had ~20k `RScan_normal` while logs showed `r_scan real_num:8` — still unexplained vs OS count; current window shows OS≈235 with many idle workers.) ### Userspace bt (valid) **TID=2283**, `comm=RScan_normal`, gdb `bt` / `bt full`: #0 futex in libc #1 pthread_cond_timedwait #2..#6 std::condition_variable::wait_for #7 doris::ThreadPool::dispatch_thread (be/src/util/threadpool.cpp:564) #8 std::function::operator() #9 doris::Thread::supervise_thread (be/src/util/thread.cpp:498) From `bt full`: - thread name = `"RScan_normal [worker]-2283"` - `task.runnable == nullptr` (idle) - waiting on pool `not_empty` condition_variable ### Conclusion from our side These OS threads **are real `ThreadPool` workers**, not inherited-name aliens. They sit idle in `ThreadPool::dispatch_thread` and **do not shrink back** toward `min_remote_scan_thread_num=8` after load (here ~235 idle workers, `active_threads=0`). Over longer runs they accumulate in batches (≈80–110 / ~10 min when heavy ADS job runs), eventually hit `pids.max` → `Could not create thread (error 11)` → BE abort. Please check `ThreadPool` grow/shrink for remote scan (`RScan_*`) and whether pool accounting (`r_scan real_num`) can diverge from live OS workers / pool recreations. Happy to collect any extra dump you need (e.g. second LWP with forced thread switch, or logs around pool create/destroy). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
