Re: Haproxy 2.2.0 segfault

[Christopher Faulet]

Le 24/07/2020 à 10:34, Sander Klein a écrit :

On 2020-07-20 21:41, Sander Klein wrote:

On 2020-07-20 19:16, Christopher Faulet wrote:

Le 20/07/2020 à 17:22, Sander Klein a écrit :

On 2020-07-20 16:38, Christopher Faulet wrote:

Could you retry with the latest 2.2 snapshot
(http://www.haproxy.org/download/2.2/src/snapshot/haproxy-ss-LATEST.tar.gz)
?


Yes, I just did. Still a segfault. Just in case the new core is
below.


Ok. Thanks to have tested. Could you share your configuration please ?
Don't forget to sanitize it if necessary.


I will sent if off list.



FYI, I've just tested with HAProxy 2.2.1 and I cannot reproduce it
anymore.



Nice (I guess :) ! I was busy and I forgot to ask you to test the 2.2.1. Thanks 
for the feedback.


--
Christopher Faulet

Re: Haproxy 2.2.0 segfault

[Sander Klein]

On 2020-07-20 21:41, Sander Klein wrote:

On 2020-07-20 19:16, Christopher Faulet wrote:

Le 20/07/2020 à 17:22, Sander Klein a écrit :

On 2020-07-20 16:38, Christopher Faulet wrote:

Could you retry with the latest 2.2 snapshot
(http://www.haproxy.org/download/2.2/src/snapshot/haproxy-ss-LATEST.tar.gz)
?


Yes, I just did. Still a segfault. Just in case the new core is 
below.



Ok. Thanks to have tested. Could you share your configuration please ?
Don't forget to sanitize it if necessary.


I will sent if off list.



FYI, I've just tested with HAProxy 2.2.1 and I cannot reproduce it 
anymore.


Greets,

Sander

0x2E78FBE8.asc
Description: application/pgp-keys


signature.asc
Description: OpenPGP digital signature

Re: Haproxy 2.2.0 segfault

[Sander Klein]

On 2020-07-20 19:16, Christopher Faulet wrote:

Le 20/07/2020 à 17:22, Sander Klein a écrit :

On 2020-07-20 16:38, Christopher Faulet wrote:

Could you retry with the latest 2.2 snapshot
(http://www.haproxy.org/download/2.2/src/snapshot/haproxy-ss-LATEST.tar.gz)
?


Yes, I just did. Still a segfault. Just in case the new core is below.


Ok. Thanks to have tested. Could you share your configuration please ?
Don't forget to sanitize it if necessary.


I will sent if off list.

Sander

0x2E78FBE8.asc
Description: application/pgp-keys


signature.asc
Description: OpenPGP digital signature

Re: Haproxy 2.2.0 segfault

[Christopher Faulet]

Le 20/07/2020 à 17:22, Sander Klein a écrit :

On 2020-07-20 16:38, Christopher Faulet wrote:

Could you retry with the latest 2.2 snapshot
(http://www.haproxy.org/download/2.2/src/snapshot/haproxy-ss-LATEST.tar.gz)
?


Yes, I just did. Still a segfault. Just in case the new core is below.

Ok. Thanks to have tested. Could you share your configuration please ? Don't 
forget to sanitize it if necessary.


--
Christopher Faulet

Re: Haproxy 2.2.0 segfault

[Sander Klein]

On 2020-07-20 16:38, Christopher Faulet wrote:

Could you retry with the latest 2.2 snapshot
(http://www.haproxy.org/download/2.2/src/snapshot/haproxy-ss-LATEST.tar.gz)
?


Yes, I just did. Still a segfault. Just in case the new core is below.

Reading symbols from haproxy...Reading symbols from 
/usr/lib/debug/.build-id/3e/19e8d25a73e3ae6245be1b59986c1249b3792b.debug...done.

done.
[New LWP 12514]
[New LWP 12516]
[New LWP 12517]
[New LWP 12515]
[Thread debugging using libthread_db enabled]
Using host libthread_db library 
"/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/haproxy -sf 4951 -Ws -f 
/etc/haproxy/conf.d -p /run/haproxy.pid -S /r'.

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x55830c624f97 in si_cs_send (cs=cs@entry=0x55830da71370) at 
include/haproxy/channel.h:128

128 include/haproxy/channel.h: No such file or directory.
[Current thread is 1 (Thread 0x7f127d86c280 (LWP 12514))]
(gdb) t a a bt full

Thread 4 (Thread 0x7f127cc3f700 (LWP 12515)):
#0  0x7f127dc77a97 in shutdown () at 
../sysdeps/unix/syscall-template.S:78

No locals.
#1  0x55830c55efcc in conn_sock_shutw (c=0x7f1270397240, 
c=0x7f1270397240, clean=1) at include/haproxy/connection.h:218

No locals.
#2  h1_shutw_conn (conn=0x7f1270397240, mode=mode@entry=CS_SHW_NORMAL) 
at src/mux_h1.c:2604

h1c = 0x7f12781f8370
__FUNCTION__ = "h1_shutw_conn"
__x = 
__l = 
__x = 
__l = 
__x = 
__l = 
__x = 
__l = 
#3  0x55830c55f129 in h1_shutw (cs=0x7f126826d670, 
mode=CS_SHW_NORMAL) at src/mux_h1.c:2593

h1s = 0x7f1278461510
h1c = 0x7f12781f8370
__FUNCTION__ = "h1_shutw"
__x = 
__l = 
__x = 
__l = 
__x = 
__l = 
__x = 
__l = 
__x = 
__l = 
__x = 
__l = 
__x = 
__l = 
__x = 
__l = 
__x = 
__l = 
__x = 
__l = 
#4  0x55830c6216d5 in cs_shutw (mode=CS_SHW_NORMAL, cs=out>) at include/haproxy/connection.h:258

No locals.
#5  stream_int_shutw_conn (si=0x7f126829ee28) at 
src/stream_interface.c:1052

cs = 
conn = 
ic = 0x7f126829eb90
oc = 
#6  0x55830c5c822b in si_shutw (si=0x7f126829ee28) at 
include/haproxy/stream_interface.h:428

--Type  for more, q to quit, c to continue without paging--c
No locals.
#7  process_stream (t=, context=0x7f126829eb80, 
state=) at src/stream.c:2264

srv = 
s = 0x7f126829eb80
sess = 
rqf_last = 
rpf_last = 
rq_prod_last = 
rq_cons_last = 
rp_cons_last = 
rp_prod_last = 
req_ana_back = 
req = 0x7f126829eb90
res = 0x7f126829ebf0
si_f = 0x7f126829ee28
si_b = 0x7f126829ee80
rate = 
#8  0x55830c687ec3 in run_tasks_from_lists 
(budgets=budgets@entry=0x7f127cc1c2dc) at src/task.c:476

process = 
tl_queues = 
t = 0x55830ebde640
budget_mask = 7 '\a'
done = 
queue = 
state = 
ctx = 
#9  0x55830c6888de in process_runnable_tasks () at src/task.c:672
tt = 0x55830c887600 
lrq = 
grq = 
t = 
max = {0, 196, 0}
max_total = 
tmp_list = 
queue = 3
max_processed = 
#10 0x55830c641d47 in run_poll_loop () at src/haproxy.c:2905
next = 
wake = 
#11 0x55830c6420e9 in run_thread_poll_loop (data=) at 
src/haproxy.c:3070

ptaf = 
ptif = 
ptdf = 
ptff = 
init_left = 0
init_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, 
__nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 
0x0, __next = 0x0}}, __size = '\000' , __align = 0}
init_cond = {__data = {{__wseq = 9, __wseq32 = {__low = 9, 
__high = 0}}, {__g1_start = 7, __g1_start32 = {__low = 7, __high = 0}}, 
__g_refs = {0, 0}, __g_size = {0, 0}, __g1_orig_size = 4, __wrefs = 0, 
__g_signals = {0, 0}}, __size = "\t\000\000\000\000\000\000\000\a", 
'\000' , "\004", '\000' , __align = 
9}
#12 0x7f127e23cfa3 in start_thread (arg=) at 
pthread_create.c:486

ret = 
pd = 
now = 
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139717379356416, 
-8923786372068320456, 140722675008942, 140722675008943, 139717379356416, 
94021338073344, 8794595559024712504, 8794593102455593784}, 
mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 
0x0, cleanup = 0x0, canceltype = 0}}}

not_first_call = 
#13 0x7f127dc764cf in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

No locals.

Thread 3 (Thread 0x7f12777fe700 (LWP 12517)):
#0  0x7f127dc767ef in epoll_wait (epfd=106, events=0x7f1268015c10, 
maxevents=200, timeout=timeout@entry=4) at 
../sysdeps/unix/sysv/linux/epoll_wait.c:30

resultvar = 18446744073709551612
 

Re: Haproxy 2.2.0 segfault

[Christopher Faulet]

Le 20/07/2020 à 16:23, Sander Klein a écrit :

In the meantime I've captured a coredump. It gives the following output:

GNU gdb (Debian 8.2.1-2+b3) 8.2.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later

This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
  .

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from haproxy...Reading symbols from
/usr/lib/debug/.build-id/d7/b1ac9548a895bfb6276bb7491ae8d396835cf0.debug...done.
done.
[New LWP 1821]
[New LWP 1819]
[New LWP 1820]
[New LWP 1822]
[Thread debugging using libthread_db enabled]
Using host libthread_db library
"/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/haproxy -sf 453 -Ws -f
/etc/haproxy/conf.d -p /run/haproxy.pid -S /ru'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x5637136d3e27 in si_cs_send (cs=cs@entry=0x7fee40d97520) at
include/haproxy/channel.h:128
128 include/haproxy/channel.h: No such file or directory.


It seems pretty close to the issue #762 
(https://github.com/haproxy/haproxy/issues/762). As suggested on GitHub, it may 
be related to the bug fixed by the commit 273aea4.


Could you retry with the latest 2.2 snapshot 
(http://www.haproxy.org/download/2.2/src/snapshot/haproxy-ss-LATEST.tar.gz) ?


--
Christopher Faulet

Re: Haproxy 2.2.0 segfault

[Sander Klein]

In the meantime I've captured a coredump. It gives the following output:

GNU gdb (Debian 8.2.1-2+b3) 8.2.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 


This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from haproxy...Reading symbols from 
/usr/lib/debug/.build-id/d7/b1ac9548a895bfb6276bb7491ae8d396835cf0.debug...done.

done.
[New LWP 1821]
[New LWP 1819]
[New LWP 1820]
[New LWP 1822]
[Thread debugging using libthread_db enabled]
Using host libthread_db library 
"/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/haproxy -sf 453 -Ws -f 
/etc/haproxy/conf.d -p /run/haproxy.pid -S /ru'.

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x5637136d3e27 in si_cs_send (cs=cs@entry=0x7fee40d97520) at 
include/haproxy/channel.h:128

128 include/haproxy/channel.h: No such file or directory.
[Current thread is 1 (Thread 0x7fee4f5d6700 (LWP 1821))]
(gdb) t a a bt full

Thread 4 (Thread 0x7fee4edd5700 (LWP 1822)):
#0  0x7fee50df5917 in sched_yield () at 
../sysdeps/unix/syscall-template.S:78

No locals.
#1  0x563713748ee5 in ha_thread_relax () at 
include/haproxy/thread.h:233

No locals.
#2  thread_harmless_till_end () at src/thread.c:58
No locals.
#3  0x5637135c4457 in thread_harmless_end () at 
include/haproxy/thread.h:261

No locals.
#4  _do_poll (p=, exp=0, wake=1) at src/ev_epoll.c:212
status = 
fd = 
count = 
updt_idx = 
wait_time = 
old_fd = 
#5  0x5637136f0b92 in run_poll_loop () at src/haproxy.c:2952
next = 
wake = 
#6  0x5637136f0f79 in run_thread_poll_loop (data=) at 
src/haproxy.c:3070

ptaf = 
ptif = 
ptdf = 
ptff = 
init_left = 0
init_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, 
__nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 
0x0, __next = 0x0}},

  __size = '\000' , __align = 0}
init_cond = {__data = {{__wseq = 9, __wseq32 = {__low = 9, 
__high = 0}}, {__g1_start = 7, __g1_start32 = {__low = 7, __high = 0}}, 
__g_refs = {0, 0}, __g_size = {0, 0},
__g1_orig_size = 4, __wrefs = 0, __g_signals = {0, 0}}, 
__size = "\t\000\000\000\000\000\000\000\a", '\000' , 
"\004", '\000' ,

  __align = 9}
#7  0x7fee513d4fa3 in start_thread (arg=) at 
pthread_create.c:486

ret = 
pd = 
now = 
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140661502072576, 
-6298750294423139148, 140728480730014, 140728480730015, 140661502072576, 
94794550343936,
6290062261562477748, 6290109760261295284}, 
mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 
0x0, cleanup = 0x0, canceltype = 0}}}

not_first_call = 
#8  0x7fee50e0e4cf in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

No locals.

Thread 3 (Thread 0x7fee4fdd7700 (LWP 1820)):
#0  0x7ffde71c8a49 in clock_gettime ()
No symbol table info available.
#1  0x7fee50e1bff6 in __GI___clock_gettime (clock_id=1, 
tp=0x7fee4fdb40e0) at ../sysdeps/unix/clock_gettime.c:115

retval = -1
sc_ret = 
vdsop = 
resultvar = 
__arg2 = 
__arg1 = 
_a2 = 
_a1 = 
sc_ret = 
vdsop = 
resultvar = 
__arg2 = 
__arg1 = 
_a2 = 
_a1 = 
#2  0x5637137361e5 in now_mono_time () at include/haproxy/time.h:523
ts = {tv_sec = 140661388919240, tv_nsec = 0}
ts = 
#3  __task_wakeup (t=0x56371727c270, root=0x563713936610 
) at src/task.c:149

No locals.
#4  0x563713673f9d in task_wakeup (f=256, t=) at 
include/haproxy/task.h:196

state = 
root = 
state = 
root = 
#5  stream_create_from_cs (cs=cs@entry=0x5637164df550) at 
src/stream.c:280

strm = 
#6  0x56371361088e in h1s_new_cs (h1s=0x563716d9c420) at 
src/mux_h1.c:492

cs = 0x5637164df550
cs = 
__FUNCTION__ = "h1s_new_cs"
__x = 
__l = 
__x = 
__l = 
__x = 
__l = 
__x = 
__l = 
__x = 
__l = 
__x = 
__l = 
__x = 
__l = 
__x = 
--Type  for more, q to quit, c to continue without paging--c
__l = 
__x = 
__l = 
__x = 
__l = 
#7  h1s_create 

Haproxy 2.2.0 segfault

[Sander Klein]

Hi,

Last Thursday I've upgraded to HAProxy 2.2.0 from Vincent Bernat's 
marvelous repository, but now I experience segfaults. I haven't 
investigated it further since I just discovered it. But, it seems 
related to reloading HAProxy with config changes.


The logs show:
Jul 20 09:51:05 lb01-a kernel: [5415518.837709] haproxy[7856]: segfault 
at 0 ip 556323bf6d1b sp 7fe5bb76b1c8 error 6 in 
haproxy[556323a57000+1a6000]
Jul 20 09:51:05 lb01-a kernel: [5415518.837721] Code: 83 e0 fe 48 f7 40 
08 fe ff ff ff 0f 84 a6 00 00 00 48 8b 48 10 f7 d2 83 e2 01 48 8b 14 d0 
89 ce 49 89 c8 83 e6 01 49 83 e0 fe <

49> 89 14 f0 f6 c2 01 74 5c 48 89 4a 0f 48 c7 40 10 00 00 00 00 48
Jul 20 09:51:05 lb01-a systemd[1]: haproxy.service: Main process exited, 
code=exited, status=139/n/a
Jul 20 09:51:05 lb01-a systemd[1]: haproxy.service: Failed with result 
'exit-code'.
Jul 20 09:51:06 lb01-a systemd[1]: haproxy.service: Service 
RestartSec=100ms expired, scheduling restart.
Jul 20 09:51:06 lb01-a systemd[1]: haproxy.service: Scheduled restart 
job, restart counter is at 11.

Jul 20 09:51:06 lb01-a systemd[1]: Stopped HAProxy Load Balancer.
Jul 20 09:51:06 lb01-a systemd[1]: Starting HAProxy Load Balancer...
Jul 20 09:51:07 lb01-a systemd[1]: Started HAProxy Load Balancer.

I'm not sure where to start with debugging. So, suggestions are welcome.

Regards,

Sander

0x2E78FBE8.asc
Description: application/pgp-keys


signature.asc
Description: OpenPGP digital signature