Made those changes, and it seems to be working properly, no segfault yet after ~2 minutes of checks. Thanks!
Steve Ruiz Manager - Hosting Operations Mirth ste...@mirth.com <ste...@mirthcorp.com> On Fri, Jan 10, 2014 at 3:06 PM, Baptiste <bed...@gmail.com> wrote: > Hi Steve, > > Could you give a try to the tcp-check and tell us if your have the same > issue. > In your backend, turn your httpchk related directives into: > option tcp-check > tcp-check send GET\ /cp/testcheck.html\ HTTP/1.0\r\n > tcp-check send \r\n > tcp-check expect string good > > Baptiste > > > On Fri, Jan 10, 2014 at 11:16 PM, Steve Ruiz <ste...@mirth.com> wrote: > > I'm experimenting with haproxy on a centos6 VM here. I found that when I > > specified a health check page (option httpchk GET /url), and that page > > didn't exist, we have a large 404 page returned, and that causes haproxy > to > > quickly segfault (seems like on the second try GET'ing and parsing the > > page). I couldn't figure out from the website where to submit a bug, so > I > > figure I'll try here first. > > > > Steps to reproduce: > > - setup http backend, with option httpchk and httpcheck expect string x. > > Make option httpchk point to a non-existent page > > - On backend server, set it up to serve large 404 response (in my case, > the > > 404 page is 186kB, as it has an inline graphic and inline css) > > - Start haproxy, and wait for it to segfault > > > > I wasn't sure exactly what was causing this at first, so I did some work > to > > narrow it down with GDB. The variable values from gdb led me to the > cause > > on my side, and hopefully can help you fix the issue. I could not make > this > > work with simply a large page for the http response - in that case, it > seems > > to work as advertised, only inspecting the response up to tune.chksize > > (default 16384 as i've left it). But if I do this with a 404, it seems > to > > kill it. Let me know what additional information you need if any. > Thanks > > and kudos for the great bit of software! > > > > > > #haproxy config: > > #--------------------------------------------------------------------- > > # Example configuration for a possible web application. See the > > # full configuration options online. > > # > > # http://haproxy.1wt.eu/download/1.4/doc/configuration.txt > > # > > #--------------------------------------------------------------------- > > > > # Help in developing config here: > > # https://www.twilio.com/engineering/2013/10/16/haproxy > > > > > > #--------------------------------------------------------------------- > > # Global settings > > #--------------------------------------------------------------------- > > global > > # to have these messages end up in /var/log/haproxy.log you will > > # need to: > > # > > # 1) configure syslog to accept network log events. This is done > > # by adding the '-r' option to the SYSLOGD_OPTIONS in > > # /etc/sysconfig/syslog > > # > > # 2) configure local2 events to go to the /var/log/haproxy.log > > # file. A line like the following can be added to > > # /etc/sysconfig/syslog > > # > > # local2.* /var/log/haproxy.log > > # > > log 127.0.0.1 local2 info > > > > chroot /var/lib/haproxy > > pidfile /var/run/haproxy.pid > > maxconn 4000 > > user haproxy > > group haproxy > > daemon > > > > #enable stats > > stats socket /tmp/haproxy.sock > > > > listen ha_stats :8088 > > balance source > > mode http > > timeout client 30000ms > > stats enable > > stats auth haproxystats:foobar > > stats uri /haproxy?stats > > > > #--------------------------------------------------------------------- > > # common defaults that all the 'listen' and 'backend' sections will > > # use if not designated in their block > > #--------------------------------------------------------------------- > > defaults > > mode http > > log global > > option httplog > > option dontlognull > > #keep persisten client connection open > > option http-server-close > > option forwardfor except 127.0.0.0/8 > > option redispatch > > # Limit number of retries - total time trying to connect = connect > > timeout * (#retries + 1) > > retries 2 > > timeout http-request 10s > > timeout queue 1m > > #timeout opening a tcp connection to server - should be shorter than > > timeout client and server > > timeout connect 3100 > > timeout client 30s > > timeout server 30s > > timeout http-keep-alive 10s > > timeout check 10s > > maxconn 3000 > > > > #--------------------------------------------------------------------- > > # main frontend which proxys to the backends > > #--------------------------------------------------------------------- > > frontend https_frontend > > bind :80 > > redirect scheme https if !{ ssl_fc } > > > > #config help: > > https://github.com/observing/balancerbattle/blob/master/haproxy.cfg > > bind *:443 ssl crt /etc/certs/mycert.pem ciphers > > RC4-SHA:AES128-SHA:AES:!ADH:!aNULL:!DH:!EDH:!eNULL > > mode http > > default_backend webapp > > > > #--------------------------------------------------------------------- > > # Main backend for web application servers > > #--------------------------------------------------------------------- > > backend webapp > > balance roundrobin > > #Insert cookie SERVERID to pin it to one leg > > cookie SERVERID insert nocache indirect > > #http check should pull url below > > option httpchk GET /cp/testcheck.html HTTP/1.0 > > #option httpchk GET /cp/testcheck.php HTTP/1.0 > > #http check should find string below in response to be considered up > > http-check expect string good > > #Define servers - inter=interval of 5s, rise 2=become avail after 2 > > successful checks, fall 3=take out after 3 fails > > server app1 app1.myco.com:80 weight 5 cookie app1 check inter 5000 > rise > > 2 fall 3 > > server app2 app2.myco.com:80 weight 5 cookie app1 check inter 5000 > rise > > 2 fall 3 > > #Example sorry server > > #server backup host.name:80 backup > > > > #--------------------------------------------------------------------- > > # Balancing for postfix / smtp > > #--------------------------------------------------------------------- > > listen smtp 10.40.60.25:25 > > option tcpka > > mode tcp > > option tcplog > > balance roundrobin > > server app1 app1.myco.com:25 check inter 20000 > > server app2 app2.myco.com:25 check inter 20000 > > > > > > GDB STack trace: > > Program received signal SIGSEGV, Segmentation fault. > > [Switching to Thread 0x7ffff7fea7c0 (LWP 6083)] > > 0x00007ffff6e22c64 in memcpy () from /lib64/libc.so.6 > > Missing separate debuginfos, use: debuginfo-install > > glibc-2.12-1.132.el6.x86_64 keyutils-libs-1.4-4.el6.x86_64 > > krb5-libs-1.10.3-10.el6_4.6.x86_64 libcom_err-1.41.12-18.el6.x86_64 > > libselinux-2.0.94-5.3.el6_4.1.x86_64 > nss-softokn-freebl-3.14.3-9.el6.x86_64 > > openssl-1.0.1e-16.el6_5.1.x86_64 pcre-7.8-6.el6.x86_64 > > zlib-1.2.3-29.el6.x86_64 > > (gdb) bt > > #0 0x00007ffff6e22c64 in memcpy () from /lib64/libc.so.6 > > #1 0x0000000000423b42 in bo_putblk (b=0x6eff90, blk=0x6d82d0 "GET > > /cp/testcheck.php HTTP/1.0\r\n", len=32) > > at include/common/buffer.h:388 > > #2 0x0000000000425963 in process_chk (t=0x70fd80) at src/checks.c:1536 > > #3 0x000000000040e7e0 in process_runnable_tasks (next=0x7fffffffe45c) at > > src/task.c:240 > > #4 0x00000000004051d0 in run_poll_loop () at src/haproxy.c:1278 > > #5 0x00000000004075f1 in main (argc=<value optimized out>, argv=<value > > optimized out>) at src/haproxy.c:1609 > > (gdb) > > > > Full GDB output: > > [root@proxy1 ~]# gdb --args /usr/sbin/haproxy -Ds -f > > /etc/haproxy/haproxy.cfg > > GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1) > > Copyright (C) 2010 Free Software Foundation, Inc. > > License GPLv3+: GNU GPL version 3 or later > > <http://gnu.org/licenses/gpl.html> > > This is free software: you are free to change and redistribute it. > > There is NO WARRANTY, to the extent permitted by law. Type "show > copying" > > and "show warranty" for details. > > This GDB was configured as "x86_64-redhat-linux-gnu". > > For bug reporting instructions, please see: > > <http://www.gnu.org/software/gdb/bugs/>... > > Reading symbols from /usr/sbin/haproxy...Reading symbols from > > /usr/lib/debug/usr/sbin/haproxy.debug...done. > > done. > > (gdb) set follow-fork-mode child > > (gdb) break src/checks.c:1536 > > Breakpoint 1 at 0x425949: file src/checks.c, line 1536. > > (gdb) run > > Starting program: /usr/sbin/haproxy -Ds -f /etc/haproxy/haproxy.cfg > > [Thread debugging using libthread_db enabled] > > [WARNING] 009/124757 (7470) : config : 'option forwardfor' ignored for > proxy > > 'smtp' as it requires HTTP mode. > > [WARNING] 009/124757 (7470) : config : 'option http-server-close' ignored > > for proxy 'smtp' as it requires HTTP mode. > > [New process 7473] > > [Thread debugging using libthread_db enabled] > > [Switching to Thread 0x7ffff7fea7c0 (LWP 7473)] > > > > Breakpoint 1, process_chk (t=0x70fd00) at src/checks.c:1536 > > 1536 bo_putblk(check->bo, s->proxy->check_req, s->proxy->check_len); > > Missing separate debuginfos, use: debuginfo-install > > glibc-2.12-1.132.el6.x86_64 keyutils-libs-1.4-4.el6.x86_64 > > krb5-libs-1.10.3-10.el6_4.6.x86_64 libcom_err-1.41.12-18.el6.x86_64 > > libselinux-2.0.94-5.3.el6_4.1.x86_64 > nss-softokn-freebl-3.14.3-9.el6.x86_64 > > openssl-1.0.1e-16.el6_5.1.x86_64 pcre-7.8-6.el6.x86_64 > > zlib-1.2.3-29.el6.x86_64 > > (gdb) continue > > Continuing. > > > > Breakpoint 1, process_chk (t=0x70fd80) at src/checks.c:1536 > > 1536 bo_putblk(check->bo, s->proxy->check_req, s->proxy->check_len); > > (gdb) continue > > Continuing. > > > > Breakpoint 1, process_chk (t=0x70fd00) at src/checks.c:1536 > > 1536 bo_putblk(check->bo, s->proxy->check_req, s->proxy->check_len); > > (gdb) continue > > Continuing. > > > > Breakpoint 1, process_chk (t=0x70fd80) at src/checks.c:1536 > > 1536 bo_putblk(check->bo, s->proxy->check_req, s->proxy->check_len); > > (gdb) step > > bo_putblk (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php HTTP/1.0\r\n", > > len=32) at include/common/buffer.h:375 > > 375 { > > (gdb) step > > 379 if (len > b->size - cur_len) > > (gdb) step > > 376 int cur_len = buffer_len(b); > > (gdb) step > > buffer_len (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php > HTTP/1.0\r\n", > > len=32) at include/common/buffer.h:151 > > 151 return buf->i + buf->o; > > (gdb) step > > bo_putblk (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php HTTP/1.0\r\n", > > len=32) at include/common/buffer.h:375 > > 375 { > > (gdb) step > > 376 int cur_len = buffer_len(b); > > (gdb) step > > buffer_len (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php > HTTP/1.0\r\n", > > len=32) at include/common/buffer.h:151 > > 151 return buf->i + buf->o; > > (gdb) step > > 379 if (len > b->size - cur_len) > > (gdb) step > > bo_putblk (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php HTTP/1.0\r\n", > > len=32) at include/common/buffer.h:380 > > 380 len = (b->size - cur_len); > > (gdb) step > > 381 if (!len) > > (gdb) step > > 384 half = buffer_contig_space(b); > > (gdb) step > > buffer_contig_space (b=<value optimized out>, blk=0x6d82d0 "GET > > /cp/testcheck.php HTTP/1.0\r\n", len=32) > > at include/common/buffer.h:222 > > 222 if (buf->data + buf->o <= buf->p) > > (gdb) step > > 223 right = buf->data + buf->size; > > (gdb) step > > 222 if (buf->data + buf->o <= buf->p) > > (gdb) step > > 223 right = buf->data + buf->size; > > (gdb) step > > 222 if (buf->data + buf->o <= buf->p) > > (gdb) step > > 227 left = buffer_wrap_add(buf, buf->p + buf->i); > > (gdb) step > > buffer_wrap_add (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php > > HTTP/1.0\r\n", len=32) at include/common/buffer.h:191 > > 191 if (ptr - buf->size >= buf->data) > > (gdb) step > > bo_putblk (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php HTTP/1.0\r\n", > > len=32) at include/common/buffer.h:388 > > 388 memcpy(b->p, blk, half); > > (gdb) print b->p > > $1 = 0x6effa4 "play:table-column;float:none}table > td[class*=\"col-\"],table > > > th[class*=\"col-\"]{display:table-cell;float:none}.table>thead>tr>td.active,.table>tbody>tr>td.active,.table>tfoot>tr>td.active,.table>thead>tr"... > > (gdb) print blk > > $2 = 0x6d82d0 "GET /cp/testcheck.php HTTP/1.0\r\n" > > (gdb) print half > > $3 = <value optimized out> > > (gdb) > > $4 = <value optimized out> > > (gdb) step > > 384 half = buffer_contig_space(b); > > (gdb) step > > buffer_contig_space (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php > > HTTP/1.0\r\n", len=32) at include/common/buffer.h:227 > > 227 left = buffer_wrap_add(buf, buf->p + buf->i); > > (gdb) step > > buffer_wrap_add (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php > > HTTP/1.0\r\n", len=32) at include/common/buffer.h:191 > > 191 if (ptr - buf->size >= buf->data) > > (gdb) step > > buffer_contig_space (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php > > HTTP/1.0\r\n", len=32) at include/common/buffer.h:228 > > 228 return right - left; > > (gdb) step > > bo_putblk (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php HTTP/1.0\r\n", > > len=32) at include/common/buffer.h:388 > > 388 memcpy(b->p, blk, half); > > (gdb) print b->p > > $5 = 0x6effa4 "play:table-column;float:none}table > td[class*=\"col-\"],table > > > th[class*=\"col-\"]{display:table-cell;float:none}.table>thead>tr>td.active,.table>tbody>tr>td.active,.table>tfoot>tr>td.active,.table>thead>tr"... > > (gdb) print blk > > $6 = 0x6d82d0 "GET /cp/testcheck.php HTTP/1.0\r\n" > > (gdb) print half > > $7 = -2119952626 > > (gdb) > > $8 = -2,119,952,626 > > (gdb) step > > > > Program received signal SIGSEGV, Segmentation fault. > > 0x00007ffff6e22c64 in memcpy () from /lib64/libc.so.6 > > > > > > > > > > Steve Ruiz > > Manager - Hosting Operations > > Mirth > > ste...@mirth.com > > > > CONFIDENTIALITY NOTICE: The information contained in this electronic > > transmission may be confidential. If you are not an intended recipient, > be > > aware that any disclosure, copying, distribution or use of the > information > > contained in this transmission is prohibited and may be unlawful. If you > > have received this transmission in error, please notify us by email reply > > and then erase it from your computer system. > -- CONFIDENTIALITY NOTICE: The information contained in this electronic transmission may be confidential. If you are not an intended recipient, be aware that any disclosure, copying, distribution or use of the information contained in this transmission is prohibited and may be unlawful. If you have received this transmission in error, please notify us by email reply and then erase it from your computer system.
