Hi William,
On Wed, Jan 01, 2014 at 12:16:29PM +0000, William Lewis wrote:
> Hi Willy,
>
> I can reproduce this on every request that contains a header that matches the
> regex of the rspidel rule.
Cool, hopefully we'll finally manage to sort it out then.
> This is the output of the debug logging in the patch you sent.
>
> 0: cur_next=+17 used=16 buf.p=0x7fd1c18eb414 buf.size=16384 buf.p=+0 buf.o=0
> buf.i=7450
> 1: old_idx=0 cur_idx=1 ptr=0x7fd1c18eb425 end=+27 next=+29 buf_end=+7433
> 1: old_idx=1 cur_idx=3 ptr=0x7fd1c18eb442 end=+38 next=+40 buf_end=+7404
> 1: old_idx=3 cur_idx=4 ptr=0x7fd1c18eb46a end=+55 next=+57 buf_end=+7364
> 2: old_idx=3 cur_idx=4 next_idx=5 used=16 delta=-57 ptr=0x7fd1c18eb46a
> next=+0 buf_end=+7307
> 1: old_idx=3 cur_idx=5 ptr=0x7fd1c18eb46a end=+27 next=+29 buf_end=+7307
> 1: old_idx=5 cur_idx=6 ptr=0x7fd1c18eb487 end=+23 next=+25 buf_end=+7278
> 1: old_idx=6 cur_idx=7 ptr=0x7fd1c18eb4a0 end=+23 next=+25 buf_end=+7253
> 1: old_idx=7 cur_idx=8 ptr=0x7fd1c18eb4b9 end=+16 next=+18 buf_end=+7228
> 1: old_idx=8 cur_idx=9 ptr=0x7fd1c18eb4cb end=+39 next=+41 buf_end=+7210
> 1: old_idx=9 cur_idx=10 ptr=0x7fd1c18eb4f4 end=+136 next=+138 buf_end=+7169
> 1: old_idx=10 cur_idx=11 ptr=0x7fd1c18eb57e end=+57 next=+59 buf_end=+7031
> 1: old_idx=11 cur_idx=12 ptr=0x7fd1c18eb5b9 end=+119 next=+121 buf_end=+6972
> 1: old_idx=12 cur_idx=13 ptr=0x7fd1c18eb632 end=+28 next=+30 buf_end=+6851
> 1: old_idx=13 cur_idx=15 ptr=0x7fd1c18eb650 end=+24 next=+26 buf_end=+6821
> 1: old_idx=15 cur_idx=16 ptr=0x7fd1c18eb66a end=+26 next=+28 buf_end=+6795
> 1: old_idx=16 cur_idx=17 ptr=0x7fd1c18eb686 end=+22 next=+24 buf_end=+6767
So... this shows me that it worked pretty well :-/
> And his is the output of "bt full" from gdb against a core dump from the
> patched build
>
>
> #0 0x000000010cbdef40 in conn_free (conn=0x7fd1c1451d80) at connection.h:520
> 520 pool_free2(pool2_connection, conn);
> (gdb) bt full
> #0 0x000000010cbdef40 in conn_free (conn=0x7fd1c1451d80) at connection.h:520
> No locals.
> #1 0x000000010cbcf132 in si_release_endpoint (si=0x7fd1c1451b08) at
> stream_interface.h:126
> conn = (struct connection *) 0x7fd1c1451d80
> appctx = (struct appctx *) 0x10cbc1113
>
> #2 0x000000010cbceb02 in http_end_txn_clean_session (s=0x7fd1c1451880) at
> src/proto_http.c:4377
> No locals.
> #3 0x000000010cbcfe48 in http_resync_states (s=0x7fd1c1451880) at
> src/proto_http.c:4766
> txn = (struct http_txn *) 0x7fd1c14518c8
> old_req_state = 33
> old_res_state = 33
> #4 0x000000010cbd611e in http_response_forward_body (s=0x7fd1c1451880,
> res=0x7fd1c144fce0, an_bit=1048576) at src/proto_http.c:6082
> tmpbuf = (struct buffer *) 0x7fd1c18eb400
This one correctly matches the buffer above.
(...)
There are two things that could be attempted now :
- try the attached patch to disable regexec() and inconditionally
perform the action. If it still crashes, then we can rule out
any libpcre bug.
- try to eliminate some unneeded options in the config below to
find the minimal set required to crash. Ideally, I'd go with
the very minimalistic config below :
global
daemon
listen external
bind :80
mode http
maxconn 1024
timeout connect 6000
timeout client 1020000
timeout server 1020000
timeout http-request 6000
option http-server-close
rspidel ^X-Frame-Options:.*
server migw :8081
If this does not crash with the config above, please try to disable some
of the settings marked "<<<<< this one" below :
> global
> daemon
> quiet
> maxconn 1024
> pidfile haproxy.pid
> log 127.0.0.1 local0 <<<<<< this one
> log 127.0.0.1 local1 notice <<<<<< this one
>
> defaults
> log global <<<<<< this one
>
> balance roundrobin <<<<<< this one
> mode http
> http-check send-state <<<<<< this one
>
> retries 3
>
> timeout connect 6000
> timeout client 1020000
> timeout server 1020000
> timeout http-request 6000
>
> option abortonclose <<<<<< this one
> option forwardfor except 127.0.0.1 <<<<<< this one
> option http-pretend-keepalive <<<<<< this one
> option http-server-close
> option httplog <<<<<< this one
> option log-health-checks <<<<<< this one
> option log-separate-errors <<<<<< this one
> option redispatch <<<<<< this one
> option tcpka <<<<<< this one
>
> errorfile 200 errorfiles/200.http <<<<<< this one
> errorfile 400 errorfiles/400.http <<<<<< this one
> errorfile 403 errorfiles/403.http <<<<<< this one
> errorfile 408 errorfiles/408.http <<<<<< this one
> errorfile 500 errorfiles/500.http <<<<<< this one
> errorfile 502 errorfiles/502.http <<<<<< this one
> errorfile 503 errorfiles/503.http <<<<<< this one
>
>
> listen stats :7000
> mode http
> stats uri /
>
> frontend external
> bind :80
>
> maxconn 1024
>
> rspidel ^X-Frame-Options:.*
>
> default_backend migw
>
> backend migw
> option httpchk GET /online <<<<<< this one
> server migw :8081 check port 48080
Thanks,
Willy
