tracy1014-hub commented on issue #65416: URL: https://github.com/apache/doris/issues/65416#issuecomment-4932671524
Hi @yiguolei — captured `/metrics` from two BEs in parallel at the same workload moment, in response to your request. Gist with the raw files + a summary: **https://gist.github.com/tracy1014-hub/6c8f05326068a446cca6403694292bc7** Files in the gist: - `doris-be-0.metrics` — experiment BE with `num_cores = 8`. `doris_be_process_thread_num = 8001`. - `doris-be-3.metrics` — control BE (default config, 32-core host). `doris_be_process_thread_num = 23502`, was just over the 22k "approaching freeze" threshold at capture time. - `SUMMARY.md` — observations. ## rs_normal pool metrics (the smoking gun) Side-by-side for the `rs_normal` pool only: | metric (rs_normal) | be-0 (`num_cores=8`) | be-3 (default) | |---|---|---| | `doris_be_thread_pool_max_threads{...rs_normal...}` | **256** | **256** | | `doris_be_thread_pool_active_threads{...rs_normal...}` | **0** | **0** | | `doris_be_thread_pool_queue_size{...rs_normal...}` | 0 | 0 | | `doris_be_thread_pool_task_execution_count_total{...rs_normal...}` | 4,798 | 15,297 | | `doris_be_thread_pool_task_wait_worker_count_total{...rs_normal...}` | 4,798 | 15,297 | | `doris_be_thread_pool_submit_failed{...rs_normal...}` | 0 | 0 | | `doris_be_process_thread_num` (whole BE) | 8,001 | 23,502 | | rs_normal count in `/proc/$PID/task/` (OS reality) | ~6,039 | ~21,392 | | OS-thread / task-count ratio | 1.26 | 1.40 | ## What this tells us 1. **The pool reports `max_threads = 256` correctly** — matches the `doris_max_remote_scanner_thread_pool_thread_num = 256` override in `be.conf`. So the config is being read into the pool's reported state on both BEs, with or without `num_cores = 8`. 2. **The pool reports `active_threads = 0`** — i.e. it thinks no workers are currently busy. `queue_size = 0`, `submit_failed = 0`. From the pool's own POV everything is healthy and idle. 3. **But `/proc/$PID/task/` shows thousands of `rs_normal` threads** actually living in the OS — ~6k on be-0, ~21k on be-3 — that the pool doesn't account for. 4. The ratio of `OS rs_normal thread count / task_execution_count_total` is ~1.26–1.40 on both BEs. Either (a) each task execution causes the pool to spawn a worker that's never reaped, or (b) there's a thread-creation path that bypasses the `task_execution_count` counter entirely. Net: the cap (`256`) is being recorded into the pool's reported state, but it isn't being consulted when threads are spawned. Some path is creating `rs_normal` worker threads without going through (or without being tracked by) the pool's bookkeeping that owns the `max_threads` cap. That's why the `256` setting is dead letter — the enforcement point exists, but the spawning path doesn't pass through it. ## num_cores = 8 vs default — still no difference Both BEs report `max_threads{rs_normal} = 256` and `active_threads{rs_normal} = 0`. The other thread pool sizes (`EvHttpServer=128`, `FragmentMgrAsyncWorkThreadPool=2048`, `ls_normal=256`, `mf_normal=128`, etc.) are also identical between the two BEs — confirming that `num_cores = 8` in 4.1.1-rc01 changes the `CpuCores` field reported to the FE but doesn't actually influence any pool's sizing. ## Live state at capture time - doris-be-0 (exp, `num_cores=8`, uptime ~1h45m): `rs_normal=6224, threads=8241`, growing at ~60/min - doris-be-3 (ctrl, default, uptime ~5h15m): `rs_normal=21392, threads=23502`, growing at ~75/min, was already past the 22k warning threshold at capture The doris-be-3 BE is still running (not yet frozen). I'm letting it run toward the ~27k freeze threshold so I can also provide the freeze-time `be.INFO` + `SHOW BACKENDS\G` with `Alive=false` as I previously offered. Let me know if you'd rather I rotate it before then and skip the freeze capture — the metrics + the earlier 4h29m log may be enough for you to find the leak path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
