Hi List,
Anyone got a idea how to debug this further?
Currently its running at 100% again, any pointers to debug the process
as its running would be appreciated.
Or should i compile again from current master and 'hope' it doesn't return?
b.t.w. truss output is as follows:
kevent(3,0x0,0,{ },200,{ 0.0 }) = 0 (0x0)
kevent(3,0x0,0,{ },200,{ 0.0 }) = 0 (0x0)
kevent(3,0x0,0,{ },200,{ 0.0 }) = 0 (0x0)
kevent(3,0x0,0,{ },200,{ 0.0 }) = 0 (0x0)
kevent(3,0x0,0,{ },200,{ 0.0 }) = 0 (0x0)
kevent(3,0x0,0,{ },200,{ 0.0 }) = 0 (0x0)
kevent(3,0x0,0,{ },200,{ 0.0 }) = 0 (0x0)
Regards,
PiBa-NL (Pieter)
Op 8-8-2018 om 22:49 schreef PiBa-NL:
Hi List,
Ive got a weird issue.. and im not sure where/how to continue digging
at the moment...
Using nbthread=1 nbproc=1, a few lua scripts, ssl offloading / http
traffic.. Only a few connections < 100...
Sometimes haproxy starts using 100% cpu usage.. After a few days..
(Makes it hard to debug..)
Currently running with version 'HA-Proxy version 1.9-dev0-48d92ee
2018/07/30'
Ive ran some commands against the haproxy socket like 'show activity',
as can be seen there are lots of loops and tasks in just a second of
time.
[2.4.3-RELEASE][root@pfsense_3]/root:
/usr/local/pkg/haproxy/haproxy_socket.sh show activity
show activity thread_id: 0
date_now: 1533754729.799405
loops: 828928664
wake_cache: 845396
wake_tasks: 827400248
wake_signal: 0
poll_exp: 828245644
poll_drop: 17451
poll_dead: 0
poll_skip: 0
fd_skip: 0
fd_lock: 0
fd_del: 0
conn_dead: 0
stream: 101147
empty_rq: 1242050
long_rq: 0
[2.4.3-RELEASE][root@pfsense_3]/root:
/usr/local/pkg/haproxy/haproxy_socket.sh show activity
show activity thread_id: 0
date_now: 1533754731.084664
loops: 829000230
wake_cache: 845398
wake_tasks: 827471812
wake_signal: 0
poll_exp: 828317210
poll_drop: 17452
poll_dead: 0
poll_skip: 0
fd_skip: 0
fd_lock: 0
fd_del: 0
conn_dead: 0
stream: 101149
empty_rq: 1242050
long_rq: 0
Other than that ive tried to attach gdb and step through / log some
functions.. it passes through.
With a gdb 'command' file like bellow i created a little log of
function breakpoints hit:
set pagination off
set height 0
set logging on
delete
rbreak haproxy.c:.
rbreak session.c:.
rbreak hlua.c:.
rbreak task.c:.
commands 1-9
cont
end
cont
Which got me a log with the following content.. as you can see it
'seems' to be looping over the same task multiple times.., which might
not even be a problem.??. the t=0x802545a60 expires and wakes up ,
expires and wakes up.?.:
Breakpoint 249, __task_queue (task=0x802545a60) at src/task.c:185
185 src/task.c: No such file or directory.
Breakpoint 253, wake_expired_tasks () at src/task.c:209
209 in src/task.c
Breakpoint 250, __task_wakeup (t=0x802545a60, root=0x8ced50
) at src/task.c:72
72 in src/task.c
Breakpoint 41, sync_poll_loop () at src/haproxy.c:2378
2378 src/haproxy.c: No such file or directory.
Breakpoint 252, process_runnable_tasks () at src/task.c:275
275 src/task.c: No such file or directory.
Breakpoint 51, session_expire_embryonic (t=0x802545a60,
context=0x8024483a0, state=513) at src/session.c:389
389 src/session.c: No such file or directory.
Breakpoint 249, __task_queue (task=0x802545a60) at src/task.c:185
185 src/task.c: No such file or directory.
Breakpoint 253, wake_expired_tasks () at src/task.c:209
209 in src/task.c
Breakpoint 250, __task_wakeup (t=0x802545a60, root=0x8ced50
) at src/task.c:72
72 in src/task.c
Breakpoint 41, sync_poll_loop () at src/haproxy.c:2378
2378 src/haproxy.c: No such file or directory.
Breakpoint 252, process_runnable_tasks () at src/task.c:275
275 src/task.c: No such file or directory.
Breakpoint 51, session_expire_embryonic (t=0x802545a60,
context=0x8024483a0, state=513) at src/session.c:389
389 src/session.c: No such file or directory.
Breakpoint 249, __task_queue (task=0x802545a60) at src/task.c:185
185 src/task.c: No such file or directory.
Breakpoint 253, wake_expired_tasks () at src/task.c:209
209 in src/task.c
Breakpoint 250, __task_wakeup (t=0x802545a60, root=0x8ced50
) at src/task.c:72
72 in src/task.c
Breakpoint 41, sync_poll_loop () at src/haproxy.c:2378
2378 src/haproxy.c: No such file or directory.
Breakpoint 252, process_runnable_tasks () at src/task.c:275
275 src/task.c: No such file or directory.
Breakpoint 51, session_expire_embryonic (t=0x802545a60,
context=0x8024483a0, state=513) at src/session.c:389
389 src/session.c: No such file or directory.
Breakpoint 249, __task_queue (task=0x802545a60) at src/task.c:185
185 src/task.c: No such file or directory.
Breakpoint 253, wake_expired_tasks () at src/task.c:209
209 in src/task.c
Breakpoint 250, __task_wakeup (t=0x802545a60, root=0x8ced50
) at src/task.c:72
72 in src/task.c
Breakpoint 41, sync_poll_loop () at src/haproxy.c:2378
2378 src/haprox
