zhang-arvin opened a new pull request, #67038: URL: https://github.com/apache/doris/pull/67038
## Proposed changes Fix #66997: RScan_normal ThreadPool idle workers never shrink, OS threads accumulate beyond max_threads and eventually crash BE. ### Root Cause The `ThreadPool::dispatch_thread()` shrink logic was only checking the shrink condition (`queue empty && num_threads > min_threads`) when `std::condition_variable::wait_for()` returned `std::cv_status::timeout`. However, on some platforms (observed in Doris cloud deployments with compute-storage separation), `pthread_cond_timedwait` may return success (`no_timeout`) even when the timeout has expired. This causes idle workers to: 1. Wake up (with `no_timeout` status) 2. Skip the shrink check (guarded by `== std::cv_status::timeout`) 3. Re-enter the idle loop 4. Never check the shrink condition again 5. Never exit Over time, remote scan jobs (`RScan_normal`) create batches of ~80-110 workers per run. These accumulate to 20k+ OS threads, far exceeding `max_threads=512`, eventually hitting cgroup `pids.max` and causing BE to abort with `Could not create thread (error 11)`. ### Fix Remove the `== std::cv_status::timeout` guard — always check the shrink condition after `wait_for()` returns. The `_queue.empty()` guard already ensures workers woken for legitimate tasks will not exit prematurely. This is consistent with the existing code comment which says: "we'll recheck the empty queue case regardless" — the comment already anticipated this issue but the code guard was too restrictive. ### Testing - Code review: the shrink condition (`_queue.empty() && _num_threads + _num_threads_pending_start > _min_threads`) is still checked before exiting, so workers woken for legitimate tasks (where `_queue` is non-empty) will continue to process tasks. - The `_min_threads` guard ensures the pool never shrinks below the configured minimum. - This change is consistent with the documented behavior of `pthread_cond_timedwait` quirks on Linux. ### Types of changes - [x] Bugfix (non-breaking change which fixes an issue) Closes #66997 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
