tracy1014-hub commented on issue #65416: URL: https://github.com/apache/doris/issues/65416#issuecomment-4932331403
Quick status update — continuing to monitor, no new findings yet, but a freeze-time capture is coming up. ## Experiment status (T+46 min on doris-be-0 with `num_cores = 8`) ``` rs_normal=2729 threads=4746 rss_kb=2878644 uptime_sec=2804 ``` Rate still ~58/min, exactly matching the control group. No change from the previous update. ## Control group snapshot | BE | `num_cores` | uptime | rs_normal | Threads | rate (rs_normal/min) | |---|---|---|---|---|---| | doris-be-0 (exp) | 8 | 46m44s | 2,729 | 4,746 | 58.3 | | doris-be-1 (ctrl) | 32 | 3h18m | 11,362 | 13,455 | 57.2 | | doris-be-2 (ctrl) | 32 | 5h18m | 15,770 | 17,866 | 49.6 | | doris-be-3 (ctrl) | 32 | 4h18m | 17,736 | 19,832 | 68.4 | ## Opportunity to capture freeze-time data (the "Path B" from my earlier offer) `doris-be-3` is on track to hit the ~27k-thread freeze threshold in roughly **1h45m** (around 15:21 CST, i.e. ~07:21 UTC), assuming the rate holds. The rotator's 6h-age threshold won't pick it up before then (its 6h mark is at 15:17, but it'd still be 5h59m at that minute — the rotator skips and the next run is 16:17, too late). I'll let `doris-be-3` run uninterrupted and capture, at the moment it freezes: - `be.INFO` from startup to freeze - `jni.log` over the same window - `SHOW BACKENDS\G` showing `Alive=false` + `HeartbeatFailureCounter > 0` - `/proc/$BE_PID/status` final `Threads=` count - thread census showing `rs_normal` as the dominant thread type I'll publish those as a follow-up here. If you'd rather I rotate `doris-be-3` before it freezes (e.g. you've already seen enough freeze signatures and want me to keep the cluster clean), say the word and I'll rotate it at the ~22k mark. If, in the meantime, there are other config levers you'd like me to try on `doris-be-0` (which is still running with `num_cores = 8` and growing linearly), I can patch them in parallel without disturbing the freeze capture on `doris-be-3`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
