tracy1014-hub commented on issue #65416: URL: https://github.com/apache/doris/issues/65416#issuecomment-4931831727
Hi @yiguolei — thanks for taking a look. Answers to both questions below. ## 1. Host CPU cores - 4 BE pods run on 4 different nodes, every node has **32 cores** at the host level (`status.capacity.cpu = 32`). - BE container cgroup limits: **8 cores** cpu, **40 GB** memory (cgroup), `mem_limit = 32G` in be.conf. - Worth flagging: when I run `SHOW BACKENDS\G` from the FE, **every BE reports `CpuCores: 32`** — i.e. the BE is reporting host CPU to the FE, not the cgroup-limited 8. If the remote scanner pool is sized/expanded based on `CpuCores`, then in a containerized env it'll be over-provisioned 4x from the start, which lines up with the runaway growth we see. ## 2. BE logs from startup → capture Captured from `doris-be-0` after **4h29m** of runtime — not yet at the freeze point, but the leak is already clearly visible. Public gist with the artifacts: **https://gist.github.com/tracy1014-hub/8df6ab002a509132d7ffbac27d0f88d4** Inside the gist: - `INSTRUCTIONS.md` — layout + how to decode the tarball - `runtime-snapshot.txt` — plain-text snapshot, browse inline without decoding anything - `be.conf` — full BE config, browse inline - `doris-issue-65416.tar.gz.b64` — base64-encoded tarball, decode with `base64 -d doris-issue-65416.tar.gz.b64 | tar xzf -` to get: - `be.INFO.log.20260710-071747` — 24 MB, full startup → capture - `be.out` — startup banner, JNI init - `be.WARNING.log.20260710-071748` - `be.gc.log.20260710-071747` — JVM GC log - `jni.log` — 3.2 MB, Paimon scanner activity via JNI. Every Paimon scan spawns a new `Thread-N` and re-installs `HadoopModule` + `CodecPool.getDecompressor` — no JVM-side scanner reuse, which lines up with `rs_normal` never shrinking. - `be.conf` and `runtime-snapshot.txt` (same as the readable inline copies above) ## Snapshot at capture time (pod age 4h29m, BE still `Alive=true`, `HeartbeatFailureCounter=0`) ``` /proc/950/status: Threads: 17192 VmRSS: 5004868 kB (~4.77 GB) VmSize: 57360836 kB thread census (top 5): 15099 rs_normal [work <-- 87% of all threads, 59x the configured 256 cap 512 brpc_arrow_flig 381 doris_be 256 ls_normal [work <-- local scanner, 256 cap is enforced here 128 brpc_light cgroup memory.current: 5.8 GB ``` So at 4h29m we already have **15,099 `rs_normal` threads** — `doris_max_remote_scanner_thread_pool_thread_num = 256` in be.conf is clearly not being honored. The pod is ~7k threads short of the freeze threshold (24k–28k) we usually see, so it would have frozen in another ~1–2h on its own. ## What I can do next If startup-to-freeze is more useful than startup-to-now, I can: 1. `kubectl patch cronjob doris-be-rotator -p '{"spec":{"suspend":true}}'` to disable the workaround 2. Let `doris-be-0` run uninterrupted until it freezes (~1–2h from now given current trajectory) 3. Capture `be.INFO` + `jni.log` at the moment of freeze + a final `SHOW BACKENDS\G` showing `Alive=false` Just say the word and I'll do it. If the current 4h29m log is enough for you to start tracing where the `rs_normal` workers are being created / why the cap isn't enforced, no need for the freeze run. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
