tracy1014-hub commented on issue #65416:
URL: https://github.com/apache/doris/issues/65416#issuecomment-4932837457
Quick status note — pstack capture is in progress, freeze-time artifacts
captured in parallel.
## pstack status
Started `gdb -batch -ex "set pagination off" -ex "thread apply all bt" -p
955` on `doris-be-3` at 14:55 CST (06:55 UTC). The BE process has 25,035
threads at attach time, so the full `thread apply all bt` is taking a while —
gdb has consumed ~7 min of CPU so far (utime=361s, stime=70s) and is still
actively walking thread stacks. Estimated 20–30 min more to finish.
Output is being piped through `gdb | gzip | base64` so the file stays at 0
bytes until gdb finishes writing. I'll publish the pstack file in a follow-up
comment as soon as it's done.
## Freeze-time artifacts captured meanwhile (while gdb is still attached)
`/proc/$BE_PID/status`:
```
Name: doris_be
State: t (tracing stop) <-- ptrace stop, gdb is attached
Pid: 955
VmSize: 67149576 kB (~64 GB)
VmRSS: 7094288 kB (~6.8 GB)
Threads: 25035
```
Thread census (top 10) from `/proc/$BE_PID/task/`:
```
22945 rs_normal [work <-- 91.6% of all threads
512 brpc_arrow_flig
378 doris_be
256 ls_normal [work <-- local scanner, capped at 256 as configured
128 brpc_light
128 brpc_heavy
128 EvHttpServer [w
64 p_normal_blocki
64 SendBatchThread
64 DownloadThreadP
```
`SHOW BACKENDS\G` from FE — `doris-be-3` row:
```
Host:
doris-be-3.doris-be-internal.paimon-prod.svc.cluster.local
LastStartTime: 2026-07-10 09:17:42
Alive: false
HeartbeatFailureCounter: 17 (climbing — gdb still attached, BE not
heartbeating)
CpuCores: 32
Memory: 40.00 GB
```
The other 3 BEs (including `doris-be-0` with `num_cores = 8`) are
`Alive=true, HeartbeatFailureCounter=0`. Cluster-wise only `doris-be-3` is
affected.
## Note on the freeze
This freeze is **gdb-induced** (the BE process is in ptrace stop while gdb
walks 25k thread stacks), not the natural leak-induced freeze at ~27k threads.
The BE crossed 25k threads during gdb attach. When gdb finishes and detaches,
the BE should SIGCONT — I'll watch whether it recovers or stays frozen.
If it recovers, I'll let it run toward 27k to capture the leak-induced
freeze state separately. If it doesn't recover (HEARTBEAT_FAIL keeps climbing
after gdb detaches), I'll delete the pod and we'll have lost the freeze-time
capture but the gdb pstack + the artifacts above are still good.
`be.INFO` (29 MB, startup → freeze), `jni.log` (4.8 MB), `be.WARNING` (107
KB), `be.gc.log` (3.3 MB), and `be.out` (881 B) have all been pulled off the
pod and will be published together with the pstack file once gdb is done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
