Well, let say this is a workaround... We'll definitively have to fix the bug ;)
Baptiste On Sat, Jan 11, 2014 at 12:24 AM, Steve Ruiz <[email protected]> wrote: > Made those changes, and it seems to be working properly, no segfault yet > after ~2 minutes of checks. Thanks! > > Steve Ruiz > Manager - Hosting Operations > Mirth > [email protected] > > > On Fri, Jan 10, 2014 at 3:06 PM, Baptiste <[email protected]> wrote: >> >> Hi Steve, >> >> Could you give a try to the tcp-check and tell us if your have the same >> issue. >> In your backend, turn your httpchk related directives into: >> option tcp-check >> tcp-check send GET\ /cp/testcheck.html\ HTTP/1.0\r\n >> tcp-check send \r\n >> tcp-check expect string good >> >> Baptiste >> >> >> On Fri, Jan 10, 2014 at 11:16 PM, Steve Ruiz <[email protected]> wrote: >> > I'm experimenting with haproxy on a centos6 VM here. I found that when >> > I >> > specified a health check page (option httpchk GET /url), and that page >> > didn't exist, we have a large 404 page returned, and that causes haproxy >> > to >> > quickly segfault (seems like on the second try GET'ing and parsing the >> > page). I couldn't figure out from the website where to submit a bug, so >> > I >> > figure I'll try here first. >> > >> > Steps to reproduce: >> > - setup http backend, with option httpchk and httpcheck expect string x. >> > Make option httpchk point to a non-existent page >> > - On backend server, set it up to serve large 404 response (in my case, >> > the >> > 404 page is 186kB, as it has an inline graphic and inline css) >> > - Start haproxy, and wait for it to segfault >> > >> > I wasn't sure exactly what was causing this at first, so I did some work >> > to >> > narrow it down with GDB. The variable values from gdb led me to the >> > cause >> > on my side, and hopefully can help you fix the issue. I could not make >> > this >> > work with simply a large page for the http response - in that case, it >> > seems >> > to work as advertised, only inspecting the response up to tune.chksize >> > (default 16384 as i've left it). But if I do this with a 404, it seems >> > to >> > kill it. Let me know what additional information you need if any. >> > Thanks >> > and kudos for the great bit of software! >> > >> > >> > #haproxy config: >> > #--------------------------------------------------------------------- >> > # Example configuration for a possible web application. See the >> > # full configuration options online. >> > # >> > # http://haproxy.1wt.eu/download/1.4/doc/configuration.txt >> > # >> > #--------------------------------------------------------------------- >> > >> > # Help in developing config here: >> > # https://www.twilio.com/engineering/2013/10/16/haproxy >> > >> > >> > #--------------------------------------------------------------------- >> > # Global settings >> > #--------------------------------------------------------------------- >> > global >> > # to have these messages end up in /var/log/haproxy.log you will >> > # need to: >> > # >> > # 1) configure syslog to accept network log events. This is done >> > # by adding the '-r' option to the SYSLOGD_OPTIONS in >> > # /etc/sysconfig/syslog >> > # >> > # 2) configure local2 events to go to the /var/log/haproxy.log >> > # file. A line like the following can be added to >> > # /etc/sysconfig/syslog >> > # >> > # local2.* /var/log/haproxy.log >> > # >> > log 127.0.0.1 local2 info >> > >> > chroot /var/lib/haproxy >> > pidfile /var/run/haproxy.pid >> > maxconn 4000 >> > user haproxy >> > group haproxy >> > daemon >> > >> > #enable stats >> > stats socket /tmp/haproxy.sock >> > >> > listen ha_stats :8088 >> > balance source >> > mode http >> > timeout client 30000ms >> > stats enable >> > stats auth haproxystats:foobar >> > stats uri /haproxy?stats >> > >> > #--------------------------------------------------------------------- >> > # common defaults that all the 'listen' and 'backend' sections will >> > # use if not designated in their block >> > #--------------------------------------------------------------------- >> > defaults >> > mode http >> > log global >> > option httplog >> > option dontlognull >> > #keep persisten client connection open >> > option http-server-close >> > option forwardfor except 127.0.0.0/8 >> > option redispatch >> > # Limit number of retries - total time trying to connect = connect >> > timeout * (#retries + 1) >> > retries 2 >> > timeout http-request 10s >> > timeout queue 1m >> > #timeout opening a tcp connection to server - should be shorter than >> > timeout client and server >> > timeout connect 3100 >> > timeout client 30s >> > timeout server 30s >> > timeout http-keep-alive 10s >> > timeout check 10s >> > maxconn 3000 >> > >> > #--------------------------------------------------------------------- >> > # main frontend which proxys to the backends >> > #--------------------------------------------------------------------- >> > frontend https_frontend >> > bind :80 >> > redirect scheme https if !{ ssl_fc } >> > >> > #config help: >> > https://github.com/observing/balancerbattle/blob/master/haproxy.cfg >> > bind *:443 ssl crt /etc/certs/mycert.pem ciphers >> > RC4-SHA:AES128-SHA:AES:!ADH:!aNULL:!DH:!EDH:!eNULL >> > mode http >> > default_backend webapp >> > >> > #--------------------------------------------------------------------- >> > # Main backend for web application servers >> > #--------------------------------------------------------------------- >> > backend webapp >> > balance roundrobin >> > #Insert cookie SERVERID to pin it to one leg >> > cookie SERVERID insert nocache indirect >> > #http check should pull url below >> > option httpchk GET /cp/testcheck.html HTTP/1.0 >> > #option httpchk GET /cp/testcheck.php HTTP/1.0 >> > #http check should find string below in response to be considered up >> > http-check expect string good >> > #Define servers - inter=interval of 5s, rise 2=become avail after 2 >> > successful checks, fall 3=take out after 3 fails >> > server app1 app1.myco.com:80 weight 5 cookie app1 check inter 5000 >> > rise >> > 2 fall 3 >> > server app2 app2.myco.com:80 weight 5 cookie app1 check inter 5000 >> > rise >> > 2 fall 3 >> > #Example sorry server >> > #server backup host.name:80 backup >> > >> > #--------------------------------------------------------------------- >> > # Balancing for postfix / smtp >> > #--------------------------------------------------------------------- >> > listen smtp 10.40.60.25:25 >> > option tcpka >> > mode tcp >> > option tcplog >> > balance roundrobin >> > server app1 app1.myco.com:25 check inter 20000 >> > server app2 app2.myco.com:25 check inter 20000 >> > >> > >> > GDB STack trace: >> > Program received signal SIGSEGV, Segmentation fault. >> > [Switching to Thread 0x7ffff7fea7c0 (LWP 6083)] >> > 0x00007ffff6e22c64 in memcpy () from /lib64/libc.so.6 >> > Missing separate debuginfos, use: debuginfo-install >> > glibc-2.12-1.132.el6.x86_64 keyutils-libs-1.4-4.el6.x86_64 >> > krb5-libs-1.10.3-10.el6_4.6.x86_64 libcom_err-1.41.12-18.el6.x86_64 >> > libselinux-2.0.94-5.3.el6_4.1.x86_64 >> > nss-softokn-freebl-3.14.3-9.el6.x86_64 >> > openssl-1.0.1e-16.el6_5.1.x86_64 pcre-7.8-6.el6.x86_64 >> > zlib-1.2.3-29.el6.x86_64 >> > (gdb) bt >> > #0 0x00007ffff6e22c64 in memcpy () from /lib64/libc.so.6 >> > #1 0x0000000000423b42 in bo_putblk (b=0x6eff90, blk=0x6d82d0 "GET >> > /cp/testcheck.php HTTP/1.0\r\n", len=32) >> > at include/common/buffer.h:388 >> > #2 0x0000000000425963 in process_chk (t=0x70fd80) at src/checks.c:1536 >> > #3 0x000000000040e7e0 in process_runnable_tasks (next=0x7fffffffe45c) >> > at >> > src/task.c:240 >> > #4 0x00000000004051d0 in run_poll_loop () at src/haproxy.c:1278 >> > #5 0x00000000004075f1 in main (argc=<value optimized out>, argv=<value >> > optimized out>) at src/haproxy.c:1609 >> > (gdb) >> > >> > Full GDB output: >> > [root@proxy1 ~]# gdb --args /usr/sbin/haproxy -Ds -f >> > /etc/haproxy/haproxy.cfg >> > GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1) >> > Copyright (C) 2010 Free Software Foundation, Inc. >> > License GPLv3+: GNU GPL version 3 or later >> > <http://gnu.org/licenses/gpl.html> >> > This is free software: you are free to change and redistribute it. >> > There is NO WARRANTY, to the extent permitted by law. Type "show >> > copying" >> > and "show warranty" for details. >> > This GDB was configured as "x86_64-redhat-linux-gnu". >> > For bug reporting instructions, please see: >> > <http://www.gnu.org/software/gdb/bugs/>... >> > Reading symbols from /usr/sbin/haproxy...Reading symbols from >> > /usr/lib/debug/usr/sbin/haproxy.debug...done. >> > done. >> > (gdb) set follow-fork-mode child >> > (gdb) break src/checks.c:1536 >> > Breakpoint 1 at 0x425949: file src/checks.c, line 1536. >> > (gdb) run >> > Starting program: /usr/sbin/haproxy -Ds -f /etc/haproxy/haproxy.cfg >> > [Thread debugging using libthread_db enabled] >> > [WARNING] 009/124757 (7470) : config : 'option forwardfor' ignored for >> > proxy >> > 'smtp' as it requires HTTP mode. >> > [WARNING] 009/124757 (7470) : config : 'option http-server-close' >> > ignored >> > for proxy 'smtp' as it requires HTTP mode. >> > [New process 7473] >> > [Thread debugging using libthread_db enabled] >> > [Switching to Thread 0x7ffff7fea7c0 (LWP 7473)] >> > >> > Breakpoint 1, process_chk (t=0x70fd00) at src/checks.c:1536 >> > 1536 bo_putblk(check->bo, s->proxy->check_req, s->proxy->check_len); >> > Missing separate debuginfos, use: debuginfo-install >> > glibc-2.12-1.132.el6.x86_64 keyutils-libs-1.4-4.el6.x86_64 >> > krb5-libs-1.10.3-10.el6_4.6.x86_64 libcom_err-1.41.12-18.el6.x86_64 >> > libselinux-2.0.94-5.3.el6_4.1.x86_64 >> > nss-softokn-freebl-3.14.3-9.el6.x86_64 >> > openssl-1.0.1e-16.el6_5.1.x86_64 pcre-7.8-6.el6.x86_64 >> > zlib-1.2.3-29.el6.x86_64 >> > (gdb) continue >> > Continuing. >> > >> > Breakpoint 1, process_chk (t=0x70fd80) at src/checks.c:1536 >> > 1536 bo_putblk(check->bo, s->proxy->check_req, s->proxy->check_len); >> > (gdb) continue >> > Continuing. >> > >> > Breakpoint 1, process_chk (t=0x70fd00) at src/checks.c:1536 >> > 1536 bo_putblk(check->bo, s->proxy->check_req, s->proxy->check_len); >> > (gdb) continue >> > Continuing. >> > >> > Breakpoint 1, process_chk (t=0x70fd80) at src/checks.c:1536 >> > 1536 bo_putblk(check->bo, s->proxy->check_req, s->proxy->check_len); >> > (gdb) step >> > bo_putblk (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php >> > HTTP/1.0\r\n", >> > len=32) at include/common/buffer.h:375 >> > 375 { >> > (gdb) step >> > 379 if (len > b->size - cur_len) >> > (gdb) step >> > 376 int cur_len = buffer_len(b); >> > (gdb) step >> > buffer_len (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php >> > HTTP/1.0\r\n", >> > len=32) at include/common/buffer.h:151 >> > 151 return buf->i + buf->o; >> > (gdb) step >> > bo_putblk (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php >> > HTTP/1.0\r\n", >> > len=32) at include/common/buffer.h:375 >> > 375 { >> > (gdb) step >> > 376 int cur_len = buffer_len(b); >> > (gdb) step >> > buffer_len (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php >> > HTTP/1.0\r\n", >> > len=32) at include/common/buffer.h:151 >> > 151 return buf->i + buf->o; >> > (gdb) step >> > 379 if (len > b->size - cur_len) >> > (gdb) step >> > bo_putblk (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php >> > HTTP/1.0\r\n", >> > len=32) at include/common/buffer.h:380 >> > 380 len = (b->size - cur_len); >> > (gdb) step >> > 381 if (!len) >> > (gdb) step >> > 384 half = buffer_contig_space(b); >> > (gdb) step >> > buffer_contig_space (b=<value optimized out>, blk=0x6d82d0 "GET >> > /cp/testcheck.php HTTP/1.0\r\n", len=32) >> > at include/common/buffer.h:222 >> > 222 if (buf->data + buf->o <= buf->p) >> > (gdb) step >> > 223 right = buf->data + buf->size; >> > (gdb) step >> > 222 if (buf->data + buf->o <= buf->p) >> > (gdb) step >> > 223 right = buf->data + buf->size; >> > (gdb) step >> > 222 if (buf->data + buf->o <= buf->p) >> > (gdb) step >> > 227 left = buffer_wrap_add(buf, buf->p + buf->i); >> > (gdb) step >> > buffer_wrap_add (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php >> > HTTP/1.0\r\n", len=32) at include/common/buffer.h:191 >> > 191 if (ptr - buf->size >= buf->data) >> > (gdb) step >> > bo_putblk (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php >> > HTTP/1.0\r\n", >> > len=32) at include/common/buffer.h:388 >> > 388 memcpy(b->p, blk, half); >> > (gdb) print b->p >> > $1 = 0x6effa4 "play:table-column;float:none}table >> > td[class*=\"col-\"],table >> > >> > th[class*=\"col-\"]{display:table-cell;float:none}.table>thead>tr>td.active,.table>tbody>tr>td.active,.table>tfoot>tr>td.active,.table>thead>tr"... >> > (gdb) print blk >> > $2 = 0x6d82d0 "GET /cp/testcheck.php HTTP/1.0\r\n" >> > (gdb) print half >> > $3 = <value optimized out> >> > (gdb) >> > $4 = <value optimized out> >> > (gdb) step >> > 384 half = buffer_contig_space(b); >> > (gdb) step >> > buffer_contig_space (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php >> > HTTP/1.0\r\n", len=32) at include/common/buffer.h:227 >> > 227 left = buffer_wrap_add(buf, buf->p + buf->i); >> > (gdb) step >> > buffer_wrap_add (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php >> > HTTP/1.0\r\n", len=32) at include/common/buffer.h:191 >> > 191 if (ptr - buf->size >= buf->data) >> > (gdb) step >> > buffer_contig_space (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php >> > HTTP/1.0\r\n", len=32) at include/common/buffer.h:228 >> > 228 return right - left; >> > (gdb) step >> > bo_putblk (b=0x6eff90, blk=0x6d82d0 "GET /cp/testcheck.php >> > HTTP/1.0\r\n", >> > len=32) at include/common/buffer.h:388 >> > 388 memcpy(b->p, blk, half); >> > (gdb) print b->p >> > $5 = 0x6effa4 "play:table-column;float:none}table >> > td[class*=\"col-\"],table >> > >> > th[class*=\"col-\"]{display:table-cell;float:none}.table>thead>tr>td.active,.table>tbody>tr>td.active,.table>tfoot>tr>td.active,.table>thead>tr"... >> > (gdb) print blk >> > $6 = 0x6d82d0 "GET /cp/testcheck.php HTTP/1.0\r\n" >> > (gdb) print half >> > $7 = -2119952626 >> > (gdb) >> > $8 = -2,119,952,626 >> > (gdb) step >> > >> > Program received signal SIGSEGV, Segmentation fault. >> > 0x00007ffff6e22c64 in memcpy () from /lib64/libc.so.6 >> > >> > >> > >> > >> > Steve Ruiz >> > Manager - Hosting Operations >> > Mirth >> > [email protected] >> > >> > CONFIDENTIALITY NOTICE: The information contained in this electronic >> > transmission may be confidential. If you are not an intended recipient, >> > be >> > aware that any disclosure, copying, distribution or use of the >> > information >> > contained in this transmission is prohibited and may be unlawful. If you >> > have received this transmission in error, please notify us by email >> > reply >> > and then erase it from your computer system. > > > > CONFIDENTIALITY NOTICE: The information contained in this electronic > transmission may be confidential. If you are not an intended recipient, be > aware that any disclosure, copying, distribution or use of the information > contained in this transmission is prohibited and may be unlawful. If you > have received this transmission in error, please notify us by email reply > and then erase it from your computer system.
