tracy1014-hub commented on issue #65416:
URL: https://github.com/apache/doris/issues/65416#issuecomment-4932250191
Hi @yiguolei — added `num_cores = 8` to `be.conf` and restarted
`doris-be-0`. Reporting back the results.
## Verification that the config took effect
`SHOW BACKENDS\G` from FE confirms the new `doris-be-0` reports `CpuCores:
8` (vs `CpuCores: 32` on the other 3 BEs that I left as a control group):
```
*************************** 1. row ***************************
Host:
doris-be-0.doris-be-internal.paimon-prod.svc.cluster.local
LastStartTime: 2026-07-10 12:49:33
Alive: true
HeartbeatFailureCounter: 0
CpuCores: 8 <-- was 32 before
Memory: 40.00 GB
*************************** 2. row ***************************
(doris-be-1, control)
CpuCores: 32 <-- unchanged
*************************** 3. row ***************************
(doris-be-3, control)
CpuCores: 32 <-- unchanged
*************************** 4. row ***************************
(doris-be-2, control)
CpuCores: 32 <-- unchanged
```
So the lever does what you expected at the report-to-FE level: `num_cores =
8` overrides the host CPU count in `CpuCores`. But the actual leak behavior is
unchanged — see below.
## rs_normal growth: experiment vs control (~30 min in)
I kept `doris-be-1/2/3` on the default config (host = 32 cores, `CpuCores:
32`) as a control group, and ran all 4 BEs in parallel under the same workload.
Side-by-side snapshot at experiment T+30 min:
| BE | `num_cores` | uptime | rs_normal | Threads | rate (rs_normal/min) |
|---|---|---|---|---|---|
| **doris-be-0** (exp) | **8** | 29m54s | 1,744 | 3,761 | **58.3** |
| doris-be-1 (ctrl) | 32 (default) | 3h01m | 10,382 | 12,475 | 57.1 |
| doris-be-2 (ctrl) | 32 (default) | 5h01m | 14,975 | 17,071 | 49.6 |
| doris-be-3 (ctrl) | 32 (default) | 4h02m | 16,571 | 18,667 | 68.4 |
Control group avg rate: **58.4 rs_normal/min**. Experiment rate: **58.3
rs_normal/min**. Essentially identical.
## rs_normal trajectory on doris-be-0 (num_cores=8) over 30 min
| pod age | rs_normal | Threads | instantaneous rate |
|---|---|---|---|
| 0m39s | 8 | 2,014 | — |
| 3m10s | 164 | 2,181 | 62/min |
| 4m06s | 224 | 2,241 | 60/min |
| 9m07s | 494 | 2,511 | 54/min |
| 14m10s | 794 | 2,811 | 60/min |
| 19m13s | 1,089 | 3,106 | 59/min |
| 24m15s | 1,389 | 3,406 | 60/min |
| 29m18s | 1,684 | 3,701 | 59/min |
Growth is linear at ~60/min and shows no sign of saturating. If the `256`
cap from `doris_max_remote_scanner_thread_pool_thread_num = 256` were being
honored, rs_normal should have stopped at 256 — instead it's at 1,684 (6.6× the
cap) and still climbing linearly.
## Observations
1. **`num_cores = 8` changes `CpuCores` but not the leak.** The hypothesis
that the rs_normal pool is sized on `CpuCores` doesn't hold — or at least
`num_cores` alone isn't the lever that bounds the remote scanner pool in
4.1.1-rc01.
2. **The 256 cap is still not enforced** under `num_cores = 8` — rs_normal
passed 256 within the first ~5 minutes and kept climbing linearly. So whatever
path is spawning rs_normal workers, it isn't consulting
`doris_max_remote_scanner_thread_pool_thread_num` at all (with or without
`num_cores`).
3. Other thread pool sizes (brpc, ls_normal, EvHttpServer, etc.) also appear
unchanged between the experiment BE and the control BEs — `num_cores = 8`
doesn't seem to affect any pool sizing I can observe.
## Cluster status
All 4 BEs are `Alive=true`, `HeartbeatFailureCounter=0`. The 3 control BEs
are at 12.5k–18.7k threads each — still below the freeze threshold (~27k) we
usually see, but doris-be-3 (highest, 18.7k at 68/min) will approach the
threshold in ~2h. I have a rotator CronJob that will pick up the oldest BE at
the 6h-age mark, and I'll manually rotate doris-be-3 before then if its thread
count crosses ~22k.
## What's next?
Standing by for direction. Things I can do on request:
- Provide `be.INFO` + `jni.log` from the experiment BE at any point if you
want to trace where the rs_normal workers are being created.
- Try other config levers (e.g.
`doris_remote_scanner_thread_pool_thread_num`,
`doris_scanner_thread_pool_queue_size`, `parallel_remote_scan_thread_num` if it
exists in 4.1.1) — let me know which to flip and what value to set.
- Let one of the control BEs freeze naturally and capture the freeze-time
log + `SHOW BACKENDS\G` showing `Alive=false` (this is what I offered in the
previous comment). With doris-be-3's current trajectory that would be in ~2h.
- Revert `num_cores = 8` if you'd prefer to keep the cluster on default
config while you investigate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
