Hello James,
2016-08-02 4:35 GMT+02:00 James Hartshorn <[email protected]>: > Hi, We’re running into segmentation faults on a new haproxy system we’re > developing. We’ve been building haproxy 1.6.7 on ubuntu 14.04.5 with > openssl,pcre, and zlib. The problem doesn’t manifest when running a single > process. Load testing is approximately 1gbps of ssl traffic from four test > servers on the internet, there are two backend servers handling it. It > seems that only processes assigned to handle the traffic die and only under > load. We have tried with Pthreads and Mutex off, but the problem remained. > In the config listed below I have omitted some other front/backends for > brevity, they are unused at present and are very simple (no ssl, no process > assignments). > > > Segmentation fault is as: > > [592869.807299] haproxy[31045]: segfault at 7f02cfca88e8 ip > 00007f02cf971eee sp 00007ffe1380d4a8 error 4 in libc-2.19.so > [7f02cf8da000+1ba000] > Can you get the coredump when it happens ? You can get it like this : ulimit -c unlimited echo '/tmp/coredump-%e.%p' > /proc/sys/kernel/core_pattern haproxy -f /your/config/file.cfg Then wait for crash to happen. Then, see coredump file in /tmp ; you can get the backtrace details like this : gdb /usr/bin/haproxy /tmp/coredump*** > bt > bt full This will give extra informations that will be very useful. Olivier > > Kernel is: "Linux hap01 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 > 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux”, Cpu is a single E5-2650 > v3, nic is an Intel X710 with i40e driver version 1.5.16 > > We are running nbproc, relevant config sections: > > ************** > global > daemon > ssl-server-verify none > log /dev/log local0 info > # log /dev/log local1 debug > # user haproxy > # group haproxy > > spread-checks 50 > #maxpipes 64000 > tune.idletimer 0 > #tune.maxpollevents 1 > #tune.comp.maxlevel 9 > #tune.zlib.memlevel 9 > stats socket /run/haproxy/stats1 uid 0 gid 0 mode 0777 level user > process 1 > stats socket /run/haproxy/stats2 uid 0 gid 0 mode 0777 level user > process 2 > stats socket /run/haproxy/stats3 uid 0 gid 0 mode 0777 level user > process 3 > stats socket /run/haproxy/stats4 uid 0 gid 0 mode 0777 level user > process 4 > stats socket /run/haproxy/stats5 uid 0 gid 0 mode 0777 level user > process 5 > stats socket /run/haproxy/stats6 uid 0 gid 0 mode 0777 level user > process 6 > stats socket /run/haproxy/stats7 uid 0 gid 0 mode 0777 level user > process 7 > stats socket /run/haproxy/stats8 uid 0 gid 0 mode 0777 level user > process 8 > #stats socket /run/haproxy/stats9 uid 0 gid 0 mode 0777 level user > process 9 > > stats bind-process all > > nbproc 8 > cpu-map 1 1 > cpu-map 2 2 > cpu-map 3 3 > cpu-map 4 4 > cpu-map 5 5 > cpu-map 6 6 > cpu-map 7 7 > cpu-map 8 8 > #cpu-map 9 9 > > maxconn 100000 > > defaults > log global > timeout server 5s > timeout connect 5s > timeout client 5s > option accept-invalid-http-request > # option http-ignore-probes > # option dontlognull > mode http > option dontlognull > option splice-request > option splice-response > default-server inter 100s > timeout connect 5000 > timeout client 50000 > timeout server 50000 > compression algo gzip > > frontend app-http > bind public.ip:80 interface p2p1 process 1-3 > bind public.ip:443 ssl crt /haproxy/ssl/_wildcard_.pem interface p2p1 > process 4-7 > option httplog > log global > mode http > acl white_list src someiprange/24 someip > tcp-request content accept if white_list > tcp-request content reject > > default_backend app-http-backend > > backend app-http-backend > bind-process 8 > mode http > option httplog > log global > option httpchk > balance static-rr > > server server1-8080 internal.ip:8082 check port 8082 > server server2-8080 internal.ip:8082 check port 8082 > > listen stats > bind internal.ip:1901 process 1 > bind internal.ip:1902 process 2 > bind internal.ip:1903 process 3 > bind internal.ip:1904 process 4 > bind internal.ip:1905 process 5 > bind internal.ip:1906 process 6 > bind internal.ip:1907 process 7 > bind internal.ip:1908 process 8 > mode http > stats enable > stats uri / > stats show-node > stats show-legends > ************************* > > Compile Line: > > make TARGET=linux2628 USE_OPENSSL=1 SSL_INC=$STATICLIBSSL/include > SSL_LIB=$STATICLIBSSL/lib ADDLIB=-ldl USE_ZLIB=1 > ZLIB_INC=/opt/zlib-$ZLIB_VERSION/ ZLIB_LIB=/opt/zlib-$ZLIB_VERSION/ > USE_STATIC_PCRE=1 PCRE_LIB=$PCRESTUFFS/lib/ PCRE_INC=$PCRESTUFFS/include/ > > Output of haproxy -vv > > ********************** > /opt/haproxy-1.6.7# ./haproxy -vv > HA-Proxy version 1.6.7 2016/07/13 > Copyright 2000-2016 Willy Tarreau <[email protected]> > > Build options : > TARGET = linux2628 > CPU = generic > CC = gcc > CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement > OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_STATIC_PCRE=1 > > Default settings : > maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 > > Encrypted password support via crypt(3): yes > Built with zlib version : 1.2.8 > Compression algorithms supported : identity("identity"), > deflate("deflate"), raw-deflate("deflate"), gzip("gzip") > Built with OpenSSL version : OpenSSL 1.0.1t 3 May 2016 > Running on OpenSSL version : OpenSSL 1.0.1t 3 May 2016 > OpenSSL library supports TLS extensions : yes > OpenSSL library supports SNI : yes > OpenSSL library supports prefer-server-ciphers : yes > Built with PCRE version : 8.38 2015-11-23 > PCRE library supports JIT : no (USE_PCRE_JIT not set) > Built without Lua support > Built with transparent proxy support using: IP_TRANSPARENT > IPV6_TRANSPARENT IP_FREEBIND > > Available polling systems : > epoll : pref=300, test result OK > poll : pref=200, test result OK > select : pref=150, test result OK > Total: 3 (3 usable), will use epoll. > ******************************* > > While I did notice a newer PCRE I haven’t built with it yet, but that > doesn’t seem to be the area of problem. I also noticed when using the > USE_PTHREAD_PSHARED=1 or USE_FUTEX= options on the make command the > “OPTION=“ output of haproxy -vv doesn’t change, though a close examination > of the output of the make indicates they are being respected. I do the > static compile of pcre, zlib, and openssl once. For each compile of > haproxy I do make clean first. > > p.s. In the Atomic Operations section of the readme: "you willy have to > either use” > >
