Hi Merton,
> I tried compiling with "make CFLAGS="-g -O0"", but it complained that
> the TARGET variable must be specified. So I compiled using:
> make CFLAGS="-g -O0" TARGET=linux2628 USE_OPENSSL=1. Funny enough, the
> compiled binary did not crash anymore. I then tried: make CFLAGS="-g
> -O0" TARGET=linux2628 USE_OPENSSL=1 USE_STATIC_PCRE=1. This time,
> request failed but the process did not crash.
Mmmh, ok. Could you give a last try with:
make clean && \
make CFLAGS="-O0 -march=native -g -fno-strict-aliasing" CPU=native \
TARGET=linux2628 USE_OPENSSL=1 USE_STATIC_PCRE=1
Any reason you use static PCRE instead of just a shared one?
> Finally, I tried compiling with the exact options that produce my
> original problem: make TARGET=linux2628 CPU=native USE_STATIC_PCRE=1
> USE_OPENSSL=1, and use gdb to trace it.
>
> Below is what I got when a request crashes the process:
>
> (gdb) run
> Starting program: /usr/bin/sudo haproxy -f /etc/haproxy/haproxy.cfg -p
> /var/run/haproxy.pid -sf 18904
> process 18911 is executing new program: /usr/local/sbin/haproxy-1.5.1
> [WARNING] 189/125516 (18911) : Setting tune.ssl.default-dh-param to
> 1024 by default, if your workload permits it you should set it to at
> least 2048. Please set a value>= 1024 to make this warning disappear.
>
> Program received signal SIGILL, Illegal instruction.
> 0x0808d9d6 in http_end_txn_clean_session (s=0x90e9e10) at
> src/proto_http.c:4835
> 4835 s->logs.bytes_in -= s->req->buf->i;
> (gdb) backtrace
> #0 0x0808d9d6 in http_end_txn_clean_session (s=0x90e9e10) at
> src/proto_http.c:4835
> #1 0x0808e0c7 in http_resync_states (s=0x90e9e10) at src/proto_http.c:5276
> #2 0x080910b8 in http_response_forward_body (s=0x90e9e10,
> res=0x90feaa0, an_bit=1048576) at src/proto_http.c:6645
> #3 0x080b05a1 in process_session (t=0x90c5cf8) at src/session.c:2053
> #4 0x080568a8 in process_runnable_tasks (next=0xbfcbbf9c) at src/task.c:238
> #5 0x0804c903 in run_poll_loop () at src/haproxy.c:1304
> #6 0x0804ef75 in main (argc=7, argv=0xbfcbc194) at src/haproxy.c:1638
The bug is in haproxy itself, I thought we may see something related to
either ssl or pcre. There is no post v1.5.1 crashfix in git, so this
is probably a new bug and its also where my knowledge ends.
Willy? This one here is crashing in 1.5.1, but did not crash in -dev26.
Merton, can you post a configuration which is less redacted than the one
from July 7th? Specifically the ACLs and some backend configuration. Not
interested in hostname and IP addresses here of course, but regexp in ACLs,
backend and server configuration, etc.
The request that is crashing haproxy, is it HTTPS on the frontend or
plain and simple HTTP?
Thanks,
Lukas
