This was the 'show activity' info Show activity: thread_id: 7 date_now: 1552497125.537000 loops: 1876310231 2198499593 29388065 2234235968 2189969792 23322503 11681489 1867345227 wake_cache: 4699089 4475087 5332367 4386659 4870234 5108383 5693172 4044835 wake_tasks: 1868654965 2192153566 21290780 2227958941 2182536918 15400586 2497632 1860594634 wake_signal: 0 0 0 0 0 0 0 0 poll_exp: 1873354056 2196628655 26623147 2232345602 2187407154 20508969 8190804 1864639469 poll_drop: 180791 105999 158230 91354 145087 154604 203869 160330 poll_dead: 0 0 0 0 0 0 0 0 poll_skip: 0 0 0 0 0 0 0 0 fd_skip: 0 0 0 0 0 0 0 0 fd_lock: 294901 536257 529154 502292 466785 487033 417977 185827 fd_del: 0 0 0 0 0 0 0 0 conn_dead: 0 0 0 0 0 0 0 0 stream: 2241864 1305507 1639446 1217533 1806315 1673571 2205727 1922870 empty_rq: 2027320 2602152 3377023 2577090 2438025 3050896 3109183 1715755 long_rq: 1300059 670921 20903418 600657 795707 435466 1735307 1079695 cpust_ms_tot: 76708 30331 75408 21508 31986 78251 78328 85689 cpust_ms_1s: 0 0 0 0 0 0 0 0 cpust_ms_15s: 32 0 117 0 178 46 108 24 avg_loop_us: 1 2 249 1 1 149 277 2
I've since restarted haproxy, so I can't query it currently for new info, but wheh it happens again, i'll try to keep it running on the slave node ;) On Thu, Mar 14, 2019 at 6:00 AM Willy Tarreau <[email protected]> wrote: > Hi Mark, > > On Wed, Mar 13, 2019 at 02:08:15PM +0100, Mark Janssen wrote: > > Hi, > > > > I've recenly switched a system over from 1.6.9, which has been running > fine > > for years, to 1.9.4. > > I've updated the configuration to use nbthread instead of nbproc, and > > cleaned up the config a lot. > > > > A few times now, however, i've seen haproxy using all available CPU on > the > > system, even when traffic is mostly idle (or when the loadbalancer isn't > > even active anymore after a failover to the 2nd node). > > That's not expected. I'm seeing a problem in your show info dump (thanks > for providing it) : > > > Run_queue: 4294967285 > > The run queue has a negative size, I have no idea how we ended up in this > state! So definitely the threads never sleep, believing they always have > something to do. Could you please also report the output of "show activity" > on the socket when this happens ? It will dump a number of per-thread > indicators. It's unlikely we'll spot the issue there but it could help > narrow it down. > > Thanks! > Willy > -- Mark Janssen -- [email protected] Unix / Linux Open-Source and Internet Consultant
