tracy1014-hub opened a new issue, #65416:
URL: https://github.com/apache/doris/issues/65416
# BE remote scanner thread pool (`rs_normal`) never shrinks — thread count
grows to 27k+ and BE freezes
## Environment
- **Doris version**: 4.1.1-rc01
- **Cluster size**: 4 BE + 3 FE, deployed via DorisOperator on Kubernetes
- **BE image**: official `apache/doris:doris-4.1.1-rc01-be` (x86_64)
- **Workload**: BE queries external tables backed by Paimon on S3 (Doris ×
Paimon catalog), mixed with normal internal OLAP queries
- **BE resources**: cpu 8 core, memory limit 40G (cgroup), `mem_limit=32G`
## Symptom
BE pod RSS climbs monotonically from ~3G to 6–7G over ~4h, then either:
- keeps growing until the pod is OOM-ish and gets killed by cgroup, **or**
- at ~27k threads the BE process freezes: it stops responding to FE
heartbeats, `SHOW BACKENDS` shows `Alive=false` with `SocketTimeoutException:
Read timed out`, CPU drops to ~1m, readiness probe times out. **Not** OOMKilled
— internal lock/resource exhaustion.
The only recovery is `kubectl delete pod` (restart the BE process).
## Misleading signals
All conventional memory metrics stay green, which made this look like a
"memory leak" for a while:
- `JVM heap used` — normal GC sawtooth, returns to baseline
- `jemalloc resident` — stable at ~2G
- `jvm non-heap` — stable at ~130M
- `mem_tracker` — stable
## Root cause
`/proc/$BE_PID/status` shows `Threads=16000–27000`, of which ~90% are
`rs_normal [worker]` threads in `S` (sleeping) state. Each thread has a 512K
stack → **7–12GB just for thread stacks**, which is what pushes RSS up.
Thread census command:
```bash
BE_PID=$(pgrep -f doris_be)
ls /proc/$BE_PID/task/ | while read t; do cat /proc/$BE_PID/task/$t/comm;
done \
| sort | uniq -c | sort -rn | head
```
Typical output on a frozen BE:
```
27012 rs_normal
48 olap_scan
24 brpc
...
```
The `rs_normal` pool (remote scanner, used for querying external catalogs
like Paimon/Hive/Iceberg) **only grows, never shrinks**. Idle keep-alive
eviction either isn't implemented or isn't effective in 4.1.1-rc01.
## Configs I tried to cap the pool (all ineffective on rc01)
| # | Where | Setting | Result |
|---|-------|---------|--------|
| 1 | `be.conf` | `doris_scanner_thread_pool_thread_num=256` | Only caps
`ls_normal` (local scanner). `rs_normal` still linear-grows. |
| 2 | `be.conf` | `doris_max_remote_scanner_thread_pool_thread_num=256` |
**No effect.** `rs_normal` keeps growing past 256. |
| 3 | FE | `ALTER WORKLOAD GROUP normal PROPERTIES
('max_remote_scan_thread_num'='256', 'min_remote_scan_thread_num'='48');` |
**No effect.** |
Related defaults on this BE:
- `doris_remote_scanner_thread_pool_thread_num=48` (initial)
- `doris_max_remote_scanner_thread_pool_thread_num=-1` (unlimited)
## Current workaround
CronJob that rotates one BE pod every 4h (`kubectl delete pod` on the oldest
ready BE, with guards: `ready BE >= 3` and `pod age > 6h`). This keeps the leak
from ever reaching the freeze threshold, but it's purely symptomatic.
YAML for the workaround is available if useful.
## Expected behavior
At least one of these should actually work on rc01:
1. `doris_max_remote_scanner_thread_pool_thread_num=N` should hard-cap
`rs_normal` size, excess idle threads should be reaped when the cap is hit and
load drops.
2. Idle `rs_normal` workers should be evicted after a keep-alive timeout
(like `ls_normal` does), so the pool shrinks back to
`doris_remote_scanner_thread_pool_thread_num` when external-scan load stops.
3. `ALTER WORKLOAD GROUP … max_remote_scan_thread_num=N` should bound
concurrent remote scans per workload group — currently appears to be ignored or
not enforced against the `rs_normal` pool.
## Reproduction
1. Fresh BE 4.1.1-rc01 with a Paimon catalog on S3.
2. Run a sustained mixed workload that issues external scans against Paimon
(a few QPS is enough; the pool grows even under modest load).
3. Within 4–8h, `Threads=` in `/proc/$BE_PID/status` crosses 20k and RSS
climbs correspondingly.
4. Eventually (usually 24k–28k threads) the BE stops answering FE heartbeats
and has to be killed externally.
Happy to provide thread dumps, `be.conf`, `SHOW BACKENDS`/`SHOW WORKLOAD
GROUPS` output, or a heap profile if it helps. Just let me know what's most
useful.
## Question
Is this a known issue in 4.1.1-rc01? Is there a fix in a later 4.1.x or
4.2.x release? If a fix exists, I'd like to upgrade instead of running the
rotator CronJob forever.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
