Thaks for the test Simon. I have marked things verified, if in the 7 day aging period you find any issues speak up here. Otherwise this can hopefully be released in a while and help everone.
** Tags removed: verification-needed verification-needed-bionic ** Tags added: verification-done verification-done-bionic -- You received this bug notification because you are a member of Ubuntu High Availability Team, which is subscribed to haproxy in Ubuntu. https://bugs.launchpad.net/bugs/1884149 Title: haproxy crashes on in __pool_get_first if unique-id-header is used Status in HAProxy: Fix Released Status in haproxy package in Ubuntu: Fix Released Status in haproxy source package in Bionic: Fix Committed Status in haproxy package in Debian: Fix Released Bug description: [Impact] * The handling of locks in haproxy led to a state that between idle http connections one could have indicated a connection was destroyed. In that case the code went on and accessed a just freed resource. As upstream puts it "It can have random implications between requests as it may lead a wrong connection's polling to be re-enabled or disabled for example, especially with threads." * Backport the fix from upstreams 1.8 stable branch [Test Case] * It is a race and might be hard to trigger. An haproxy config to be in front of three webservers can be seen below. Setting up three apaches locally didn't trigger the same bug, but we know it is timing sensitive. * Simon (anbox) has a setup which reliably triggers this and will run the tests there. * The bad case will trigger a crash as reported below. [Regression Potential] * This change is in >=Disco and has no further bugs reported against it (no follow on change) which should make it rather safe. Also no other change to that file context in 1.8 stable since then. The change is on the locking of connections. So if we want to expect regressions, then they would be at the handling of concurrent connections. [Other Info] * Strictly speaking it is a race, so triggering it depends on load and machine cpu/IO speed. --- Version 1.8.8-1ubuntu0.10 of haproxy in Ubuntu 18.04 (bionic) crashes with ------------------------------------ Thread 2.1 "haproxy" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xfffff77b1010 (LWP 17174)] __pool_get_first (pool=0xaaaaaac6ddd0, pool=0xaaaaaac6ddd0) at include/common/memory.h:124 124 include/common/memory.h: No such file or directory. (gdb) bt #0 __pool_get_first (pool=0xaaaaaac6ddd0, pool=0xaaaaaac6ddd0) at include/common/memory.h:124 #1 pool_alloc_dirty (pool=0xaaaaaac6ddd0) at include/common/memory.h:154 #2 pool_alloc (pool=0xaaaaaac6ddd0) at include/common/memory.h:229 #3 conn_new () at include/proto/connection.h:655 #4 cs_new (conn=0x0) at include/proto/connection.h:683 #5 connect_conn_chk (t=0xaaaaaacb8820) at src/checks.c:1553 #6 process_chk_conn (t=0xaaaaaacb8820) at src/checks.c:2135 #7 process_chk (t=0xaaaaaacb8820) at src/checks.c:2281 #8 0x0000aaaaaabca0b4 in process_runnable_tasks () at src/task.c:231 #9 0x0000aaaaaab76f44 in run_poll_loop () at src/haproxy.c:2399 #10 run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:2461 #11 0x0000aaaaaaad79ec in main (argc=<optimized out>, argv=0xaaaaaac61b30) at src/haproxy.c:3050 ------------------------------------ when running on an ARM64 system. The haproxy.cfg looks like this: ------------------------------------ global log /dev/log local0 log /dev/log local1 notice maxconn 4096 user haproxy group haproxy spread-checks 0 tune.ssl.default-dh-param 1024 ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:!DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:!DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:!CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA defaults log global mode tcp option httplog option dontlognull retries 3 timeout queue 20000 timeout client 50000 timeout connect 5000 timeout server 50000 frontend anbox-stream-gateway-lb-5-80 bind 0.0.0.0:80 default_backend api_http mode http http-request redirect scheme https backend api_http mode http frontend anbox-stream-gateway-lb-5-443 bind 0.0.0.0:443 ssl crt /var/lib/haproxy/default.pem no-sslv3 default_backend app-anbox-stream-gateway mode http backend app-anbox-stream-gateway mode http balance leastconn server anbox-stream-gateway-0-4000 10.212.218.61:4000 check ssl verify none inter 2000 rise 2 fall 5 maxconn 4096 server anbox-stream-gateway-1-4000 10.212.218.93:4000 check ssl verify none inter 2000 rise 2 fall 5 maxconn 4096 server anbox-stream-gateway-2-4000 10.212.218.144:4000 check ssl verify none inter 2000 rise 2 fall 5 maxconn 4096 ------------------------------------ The crash occurs after a first few HTTP requests going through and happens again when systemd restarts the service. The bug is already reported in Debian https://bugs.debian.org/cgi- bin/bugreport.cgi?bug=921981 and upstream at https://github.com/haproxy/haproxy/issues/40 Using the 1.8.19-1+deb10u2 package from Debian fixes the crash. To manage notifications about this bug go to: https://bugs.launchpad.net/haproxy/+bug/1884149/+subscriptions _______________________________________________ Mailing list: https://launchpad.net/~ubuntu-ha Post to : ubuntu-ha@lists.launchpad.net Unsubscribe : https://launchpad.net/~ubuntu-ha More help : https://help.launchpad.net/ListHelp
