Hi Steve, Could you give a try to the tcp-check and tell us if your have the same issue. In your backend, turn your httpchk related directives into: option tcp-check tcp-check send GET\ /cp/testcheck.html\ HTTP/1.0\r\n tcp-check send \r\n tcp-check expect string good
Baptiste On Fri, Jan 10, 2014 at 11:16 PM, Steve Ruiz <ste...@mirth.com> wrote: > I'm experimenting with haproxy on a centos6 VM here. I found that when I > specified a health check page (option httpchk GET /url), and that page > didn't exist, we have a large 404 page returned, and that causes haproxy to > quickly segfault (seems like on the second try GET'ing and parsing the > page). I couldn't figure out from the website where to submit a bug, so I > figure I'll try here first. > > Steps to reproduce: > - setup http backend, with option httpchk and httpcheck expect string x. > Make option httpchk point to a non-existent page > - On backend server, set it up to serve large 404 response (in my case, the > 404 page is 186kB, as it has an inline graphic and inline css) > - Start haproxy, and wait for it to segfault > > I wasn't sure exactly what was causing this at first, so I did some work to > narrow it down with GDB. The variable values from gdb led me to the cause > on my side, and hopefully can help you fix the issue. I could not make this > work with simply a large page for the http response - in that case, it seems > to work as advertised, only inspecting the response up to tune.chksize > (default 16384 as i've left it). But if I do this with a 404, it seems to > kill it. Let me know what additional information you need if any. Thanks > and kudos for the great bit of software! > > > #haproxy config: > #--------------------------------------------------------------------- > # Example configuration for a possible web application. See the > # full configuration options online. > # > # http://haproxy.1wt.eu/download/1.4/doc/configuration.txt > # > #--------------------------------------------------------------------- > > # Help in developing config here: > # https://www.twilio.com/engineering/2013/10/16/haproxy > > > #--------------------------------------------------------------------- > # Global settings > #--------------------------------------------------------------------- > global > # to have these messages end up in /var/log/haproxy.log you will > # need to: > # > # 1) configure syslog to accept network log events. This is done > # by adding the '-r' option to the SYSLOGD_OPTIONS in > # /etc/sysconfig/syslog > # > # 2) configure local2 events to go to the /var/log/haproxy.log > # file. A line like the following can be added to > # /etc/sysconfig/syslog > # > # local2.* /var/log/haproxy.log > # > log 127.0.0.1 local2 info > > chroot /var/lib/haproxy > pidfile /var/run/haproxy.pid > maxconn 4000 > user haproxy > group haproxy > daemon > > #enable stats > stats socket /tmp/haproxy.sock > > listen ha_stats :8088 > balance source > mode http > timeout client 30000ms > stats enable > stats auth haproxystats:foobar > stats uri /haproxy?stats > > #--------------------------------------------------------------------- > # common defaults that all the 'listen' and 'backend' sections will > # use if not designated in their block > #--------------------------------------------------------------------- > defaults > mode http > log global > option httplog > option dontlognull > #keep persisten client connection open > option http-server-close > option forwardfor except 127.0.0.0/8 > option redispatch > # Limit number of retries - total time trying to connect = connect > timeout * (#retries + 1) > retries 2 > timeout http-request 10s > timeout queue 1m > #timeout opening a tcp connection to server - should be shorter than > timeout client and server > timeout connect 3100 > timeout client 30s > timeout server 30s > timeout http-keep-alive 10s > timeout check 10s > maxconn 3000 > > #--------------------------------------------------------------------- > # main frontend which proxys to the backends > #--------------------------------------------------------------------- > frontend https_frontend > bind :80 > redirect scheme https if !{ ssl_fc } > > #config help: > https://github.com/observing/balancerbattle/blob/master/haproxy.cfg > bind *:443 ssl crt /etc/certs/mycert.pem ciphers > RC4-SHA:AES128-SHA:AES:!ADH:!aNULL:!DH:!EDH:!eNULL > mode http > default_backend webapp > > #--------------------------------------------------------------------- > # Main backend for web application servers > #--------------------------------------------------------------------- > backend webapp > balance roundrobin > #Insert cookie SERVERID to pin it to one leg > cookie SERVERID insert nocache indirect > #http check should pull url below > option httpchk GET /cp/testcheck.html HTTP/1.0 > #option httpchk GET /cp/testcheck.php HTTP/1.0 > #http check should find string below in response to be considered up > http-check expect string good > #Define servers - inter=interval of 5s, rise 2=become avail after 2 > successful checks, fall 3=take out after 3 fails > server app1 app1.myco.com:80 weight 5 cookie app1 check inter 5000 rise > 2 fall 3 > server app2 app2.myco.com:80 weight 5 cookie app1 check inter 5000 rise > 2 fall 3 > #Example sorry server > #server backup host.name:80 backup > > #--------------------------------------------------------------------- > # Balancing for postfix / smtp > #--------------------------------------------------------------------- > listen smtp 10.40.60.25:25 > option tcpka > mode tcp > option tcplog > balance roundrobin > server app1 app1.myco.com:25 check inter 20000 > server app2 app2.myco.com:25 check inter 20000 > > > GDB STack trace: > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 0x7ffff7fea7c0 (LWP 6083)] > 0x00007ffff6e22c64 in memcpy () from /lib64/libc.so.6 > Missing separate debuginfos, use: debuginfo-install > glibc-2.12-1.132.el6.x86_64 keyutils-libs-1.4-4.el6.x86_64 > krb5-libs-1.10.3-10.el6_4.6.x86_64 libcom_err-1.41.12-18.el6.x86_64 > libselinux-2.0.94-5.3.el6_4.1.x86_64 nss-softokn-freebl-3.14.3-9.el6.x86_64 > openssl-1.0.1e-16.el6_5.1.x86_64 pcre-7.8-6.el6.x86_64 > zlib-1.2.3-29.el6.x86_64 > (gdb) bt > #0 0x00007ffff6e22c64 in memcpy () from /lib64/libc.so.6 > #1 0x0000000000423b42 in bo_putblk (b=0x6eff90, blk=0x6d82d0 "GET > /cp/testcheck.php HTTP/1.0\r\n", len=32) > at include/common/buffer.h:388 > #2 0x0000000000425963 in process_chk (t=0x70fd80) at src/checks.c:1536 > #3 0x000000000040e7e0 in process_runnable_tasks (next=0x7fffffffe45c) at > src/task.c:240 > #4 0x00000000004051d0 in run_poll_loop () at src/haproxy.c:1278 > #5 0x00000000004075f1 in main (argc=<value optimized out>, argv=<value > optimized out>) at src/haproxy.c:1609 > (gdb) > > Full GDB output: > [root@proxy1 ~]# gdb --args /usr/sbin/haproxy -Ds -f > /etc/haproxy/haproxy.cfg > GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1) > Copyright (C) 2010 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later > <http://gnu.org/licenses/gpl.html> > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > and "show warranty" for details. > This GDB was configured as "x86_64-redhat-linux-gnu". > For bug reporting instructions, please see: > <http://www.gnu.org/software/gdb/bugs/>... > Reading symbols from /usr/sbin/haproxy...Reading symbols from > /usr/lib/debug/usr/sbin/haproxy.debug...done. > done. > (gdb) set follow-fork-mode child > (gdb) break src/checks.c:1536 > Breakpoint 1 at 0x425949: file src/checks.c, line 1536. > (gdb) run > Starting program: /usr/sbin/haproxy -Ds -f /etc/haproxy/haproxy.cfg > [Thread debugging using libthread_db enabled] > [WARNING] 009/124757 (7470) : config : 'option forwardfor' ignored for proxy > 'smtp' as it requires HTTP mode. > [WARNING] 009/124757 (7470) : config : 'option http-server-close' ignored > for proxy 'smtp' as it requires HTTP mode. > [New process 7473] > [Thread debugging using libthread_db enabled] > [Switching to Thread 0x7ffff7fea7c0 (LWP 7473)] > > Breakpoint 1, process_chk (t=0x70fd00) at src/checks.c:1536 > 1536 bo_putblk(check->bo, s->proxy->check_req, s->proxy->check_len); > Missing separate debuginfos, use: debuginfo-install > glibc-2.12-1.132.el6.x86_64 keyutils-libs-1.4-4.el6.x86_64 > krb5-libs-1.10.3-10.el6_4.6.x86_64 libcom_err-1.41.12-18.el6.x86_64 > libselinux-2.0.94-5.3.el6_4.1.x86_64 nss-softokn-freebl-3.14.3-9.el6.x86_64 > openssl-1.0.1e-16.el6_5.1.x86_64 pcre-7.8-6.el6.x86_64 > zlib-1.2.3-29.el6.x86_64 > (gdb) continue > Continuing. > > Breakpoint 1, process_chk (t=0x70fd80) at src/checks.c:1536 > 1536 bo_putblk(check->bo, s->proxy->check_req, s->proxy->check_len); > (gdb) continue > Continuing. > > Breakpoint 1, process_chk (t=0x70fd00) at src/checks.c:1536 > 1536 bo_putblk(check->bo, s->proxy->check_req, s->proxy->check_len); > (gdb) continue > Continuing. > > Breakpoint 1, process_chk (t=0x70fd80) at src/checks.c:1536 > 1536 bo_putblk(check->bo, s->proxy->check_req, s->proxy->check_len); > (gdb) step > bo_putblk (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php HTTP/1.0\r\n", > len=32) at include/common/buffer.h:375 > 375 { > (gdb) step > 379 if (len > b->size - cur_len) > (gdb) step > 376 int cur_len = buffer_len(b); > (gdb) step > buffer_len (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php HTTP/1.0\r\n", > len=32) at include/common/buffer.h:151 > 151 return buf->i + buf->o; > (gdb) step > bo_putblk (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php HTTP/1.0\r\n", > len=32) at include/common/buffer.h:375 > 375 { > (gdb) step > 376 int cur_len = buffer_len(b); > (gdb) step > buffer_len (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php HTTP/1.0\r\n", > len=32) at include/common/buffer.h:151 > 151 return buf->i + buf->o; > (gdb) step > 379 if (len > b->size - cur_len) > (gdb) step > bo_putblk (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php HTTP/1.0\r\n", > len=32) at include/common/buffer.h:380 > 380 len = (b->size - cur_len); > (gdb) step > 381 if (!len) > (gdb) step > 384 half = buffer_contig_space(b); > (gdb) step > buffer_contig_space (b=<value optimized out>, blk=0x6d82d0 "GET > /cp/testcheck.php HTTP/1.0\r\n", len=32) > at include/common/buffer.h:222 > 222 if (buf->data + buf->o <= buf->p) > (gdb) step > 223 right = buf->data + buf->size; > (gdb) step > 222 if (buf->data + buf->o <= buf->p) > (gdb) step > 223 right = buf->data + buf->size; > (gdb) step > 222 if (buf->data + buf->o <= buf->p) > (gdb) step > 227 left = buffer_wrap_add(buf, buf->p + buf->i); > (gdb) step > buffer_wrap_add (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php > HTTP/1.0\r\n", len=32) at include/common/buffer.h:191 > 191 if (ptr - buf->size >= buf->data) > (gdb) step > bo_putblk (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php HTTP/1.0\r\n", > len=32) at include/common/buffer.h:388 > 388 memcpy(b->p, blk, half); > (gdb) print b->p > $1 = 0x6effa4 "play:table-column;float:none}table td[class*=\"col-\"],table > th[class*=\"col-\"]{display:table-cell;float:none}.table>thead>tr>td.active,.table>tbody>tr>td.active,.table>tfoot>tr>td.active,.table>thead>tr"... > (gdb) print blk > $2 = 0x6d82d0 "GET /cp/testcheck.php HTTP/1.0\r\n" > (gdb) print half > $3 = <value optimized out> > (gdb) > $4 = <value optimized out> > (gdb) step > 384 half = buffer_contig_space(b); > (gdb) step > buffer_contig_space (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php > HTTP/1.0\r\n", len=32) at include/common/buffer.h:227 > 227 left = buffer_wrap_add(buf, buf->p + buf->i); > (gdb) step > buffer_wrap_add (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php > HTTP/1.0\r\n", len=32) at include/common/buffer.h:191 > 191 if (ptr - buf->size >= buf->data) > (gdb) step > buffer_contig_space (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php > HTTP/1.0\r\n", len=32) at include/common/buffer.h:228 > 228 return right - left; > (gdb) step > bo_putblk (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php HTTP/1.0\r\n", > len=32) at include/common/buffer.h:388 > 388 memcpy(b->p, blk, half); > (gdb) print b->p > $5 = 0x6effa4 "play:table-column;float:none}table td[class*=\"col-\"],table > th[class*=\"col-\"]{display:table-cell;float:none}.table>thead>tr>td.active,.table>tbody>tr>td.active,.table>tfoot>tr>td.active,.table>thead>tr"... > (gdb) print blk > $6 = 0x6d82d0 "GET /cp/testcheck.php HTTP/1.0\r\n" > (gdb) print half > $7 = -2119952626 > (gdb) > $8 = -2,119,952,626 > (gdb) step > > Program received signal SIGSEGV, Segmentation fault. > 0x00007ffff6e22c64 in memcpy () from /lib64/libc.so.6 > > > > > Steve Ruiz > Manager - Hosting Operations > Mirth > ste...@mirth.com > > CONFIDENTIALITY NOTICE: The information contained in this electronic > transmission may be confidential. If you are not an intended recipient, be > aware that any disclosure, copying, distribution or use of the information > contained in this transmission is prohibited and may be unlawful. If you > have received this transmission in error, please notify us by email reply > and then erase it from your computer system.
