** Description changed: - Version 1.8.8-1ubuntu0.10 of haproxy in Ubuntu 18.04 (bionic) crashes - with + [Impact] + + * The handling of locks in haproxy led to a state that between idle http + connections one could have indicated a connection was destroyed. In that + case the code went on and accessed a just freed resource. As upstream + puts it "It can have random implications between requests as + it may lead a wrong connection's polling to be re-enabled or disabled + for example, especially with threads." + + * Backport the fix from upstreams 1.8 stable branch + + [Test Case] + + * It is a race and might be hard to trigger. + An haproxy config to be in front of three webservers can be seen below. + Setting up three apaches locally didn't trigger the same bug, but we + know it is timing sensitive. + + * Simon (anbox) has a setup which reliably triggers this and will run the + tests there. + + * The bad case will trigger a crash as reported below. + + [Regression Potential] + + * This change is in >=Disco and has no further bugs reported against it + (no follow on change) which should make it rather safe. Also no other + change to that file context in 1.8 stable since then. + The change is on the locking of connections. So if we want to expect + regressions, then they would be at the handling of concurrent + connections. + + [Other Info] + + * Strictly speaking it is a race, so triggering it depends on load and + machine cpu/IO speed. + + + --- + + + Version 1.8.8-1ubuntu0.10 of haproxy in Ubuntu 18.04 (bionic) crashes with ------------------------------------ Thread 2.1 "haproxy" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xfffff77b1010 (LWP 17174)] __pool_get_first (pool=0xaaaaaac6ddd0, pool=0xaaaaaac6ddd0) at include/common/memory.h:124 124 include/common/memory.h: No such file or directory. (gdb) bt #0 __pool_get_first (pool=0xaaaaaac6ddd0, pool=0xaaaaaac6ddd0) at include/common/memory.h:124 #1 pool_alloc_dirty (pool=0xaaaaaac6ddd0) at include/common/memory.h:154 #2 pool_alloc (pool=0xaaaaaac6ddd0) at include/common/memory.h:229 #3 conn_new () at include/proto/connection.h:655 #4 cs_new (conn=0x0) at include/proto/connection.h:683 #5 connect_conn_chk (t=0xaaaaaacb8820) at src/checks.c:1553 #6 process_chk_conn (t=0xaaaaaacb8820) at src/checks.c:2135 #7 process_chk (t=0xaaaaaacb8820) at src/checks.c:2281 #8 0x0000aaaaaabca0b4 in process_runnable_tasks () at src/task.c:231 #9 0x0000aaaaaab76f44 in run_poll_loop () at src/haproxy.c:2399 #10 run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:2461 #11 0x0000aaaaaaad79ec in main (argc=<optimized out>, argv=0xaaaaaac61b30) at src/haproxy.c:3050 ------------------------------------ when running on an ARM64 system. The haproxy.cfg looks like this: ------------------------------------ global - log /dev/log local0 - log /dev/log local1 notice - maxconn 4096 - user haproxy - group haproxy - spread-checks 0 - tune.ssl.default-dh-param 1024 - ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:!DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:!DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:!CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA + log /dev/log local0 + log /dev/log local1 notice + maxconn 4096 + user haproxy + group haproxy + spread-checks 0 + tune.ssl.default-dh-param 1024 + ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:!DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:!DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:!CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA defaults - log global - mode tcp - option httplog - option dontlognull - retries 3 - timeout queue 20000 - timeout client 50000 - timeout connect 5000 - timeout server 50000 - + log global + mode tcp + option httplog + option dontlognull + retries 3 + timeout queue 20000 + timeout client 50000 + timeout connect 5000 + timeout server 50000 frontend anbox-stream-gateway-lb-5-80 - bind 0.0.0.0:80 - default_backend api_http - mode http - http-request redirect scheme https + bind 0.0.0.0:80 + default_backend api_http + mode http + http-request redirect scheme https backend api_http - mode http + mode http frontend anbox-stream-gateway-lb-5-443 - bind 0.0.0.0:443 ssl crt /var/lib/haproxy/default.pem no-sslv3 - default_backend app-anbox-stream-gateway - mode http + bind 0.0.0.0:443 ssl crt /var/lib/haproxy/default.pem no-sslv3 + default_backend app-anbox-stream-gateway + mode http backend app-anbox-stream-gateway - mode http - balance leastconn - server anbox-stream-gateway-0-4000 10.212.218.61:4000 check ssl verify none inter 2000 rise 2 fall 5 maxconn 4096 - server anbox-stream-gateway-1-4000 10.212.218.93:4000 check ssl verify none inter 2000 rise 2 fall 5 maxconn 4096 - server anbox-stream-gateway-2-4000 10.212.218.144:4000 check ssl verify none inter 2000 rise 2 fall 5 maxconn 4096 + mode http + balance leastconn + server anbox-stream-gateway-0-4000 10.212.218.61:4000 check ssl verify none inter 2000 rise 2 fall 5 maxconn 4096 + server anbox-stream-gateway-1-4000 10.212.218.93:4000 check ssl verify none inter 2000 rise 2 fall 5 maxconn 4096 + server anbox-stream-gateway-2-4000 10.212.218.144:4000 check ssl verify none inter 2000 rise 2 fall 5 maxconn 4096 ------------------------------------ The crash occurs after a first few HTTP requests going through and happens again when systemd restarts the service. The bug is already reported in Debian https://bugs.debian.org/cgi- bin/bugreport.cgi?bug=921981 and upstream at https://github.com/haproxy/haproxy/issues/40 Using the 1.8.19-1+deb10u2 package from Debian fixes the crash.
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1884149 Title: haproxy crashes on in __pool_get_first if unique-id-header is used To manage notifications about this bug go to: https://bugs.launchpad.net/haproxy/+bug/1884149/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
