This looks good, thanks!
One minor note. The dep3 header says: Origin: upstream,
https://github.com/haproxy/haproxy/commit/d9a130e1962c2a5352f33088c563f4248a102c48;
however that link says "This commit does not belong to any branch on
this repository." so it isn't technically "upstream". However, it also
says "cherry picked from commit ad7f0ad" and
https://github.com/haproxy/haproxy/commit/ad7f0ad1c3c9c541a4c315b24d4500405d1383ee
*is* upstream and clearly is the same change, reported as present in the
upstream master branch since what looks like their v1.9-dev2 tag. So
everything is fine after all. Recording this here in case the d9a130
commit disappears and someone wants to track it down.
** Changed in: haproxy (Ubuntu Bionic)
Status: Triaged => Fix Committed
** Tags added: verification-needed verification-needed-bionic
--
You received this bug notification because you are a member of Ubuntu
High Availability Team, which is subscribed to haproxy in Ubuntu.
https://bugs.launchpad.net/bugs/1884149
Title:
haproxy crashes on in __pool_get_first if unique-id-header is used
Status in HAProxy:
Fix Released
Status in haproxy package in Ubuntu:
Fix Released
Status in haproxy source package in Bionic:
Fix Committed
Status in haproxy package in Debian:
Fix Released
Bug description:
[Impact]
* The handling of locks in haproxy led to a state that between idle http
connections one could have indicated a connection was destroyed. In that
case the code went on and accessed a just freed resource. As upstream
puts it "It can have random implications between requests as
it may lead a wrong connection's polling to be re-enabled or disabled
for example, especially with threads."
* Backport the fix from upstreams 1.8 stable branch
[Test Case]
* It is a race and might be hard to trigger.
An haproxy config to be in front of three webservers can be seen below.
Setting up three apaches locally didn't trigger the same bug, but we
know it is timing sensitive.
* Simon (anbox) has a setup which reliably triggers this and will run the
tests there.
* The bad case will trigger a crash as reported below.
[Regression Potential]
* This change is in >=Disco and has no further bugs reported against it
(no follow on change) which should make it rather safe. Also no other
change to that file context in 1.8 stable since then.
The change is on the locking of connections. So if we want to expect
regressions, then they would be at the handling of concurrent
connections.
[Other Info]
* Strictly speaking it is a race, so triggering it depends on load and
machine cpu/IO speed.
---
Version 1.8.8-1ubuntu0.10 of haproxy in Ubuntu 18.04 (bionic) crashes with
------------------------------------
Thread 2.1 "haproxy" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xfffff77b1010 (LWP 17174)]
__pool_get_first (pool=0xaaaaaac6ddd0, pool=0xaaaaaac6ddd0) at
include/common/memory.h:124
124 include/common/memory.h: No such file or directory.
(gdb) bt
#0 __pool_get_first (pool=0xaaaaaac6ddd0, pool=0xaaaaaac6ddd0) at
include/common/memory.h:124
#1 pool_alloc_dirty (pool=0xaaaaaac6ddd0) at include/common/memory.h:154
#2 pool_alloc (pool=0xaaaaaac6ddd0) at include/common/memory.h:229
#3 conn_new () at include/proto/connection.h:655
#4 cs_new (conn=0x0) at include/proto/connection.h:683
#5 connect_conn_chk (t=0xaaaaaacb8820) at src/checks.c:1553
#6 process_chk_conn (t=0xaaaaaacb8820) at src/checks.c:2135
#7 process_chk (t=0xaaaaaacb8820) at src/checks.c:2281
#8 0x0000aaaaaabca0b4 in process_runnable_tasks () at src/task.c:231
#9 0x0000aaaaaab76f44 in run_poll_loop () at src/haproxy.c:2399
#10 run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:2461
#11 0x0000aaaaaaad79ec in main (argc=<optimized out>, argv=0xaaaaaac61b30) at
src/haproxy.c:3050
------------------------------------
when running on an ARM64 system. The haproxy.cfg looks like this:
------------------------------------
global
log /dev/log local0
log /dev/log local1 notice
maxconn 4096
user haproxy
group haproxy
spread-checks 0
tune.ssl.default-dh-param 1024
ssl-default-bind-ciphers
ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:!DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:!DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:!CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
defaults
log global
mode tcp
option httplog
option dontlognull
retries 3
timeout queue 20000
timeout client 50000
timeout connect 5000
timeout server 50000
frontend anbox-stream-gateway-lb-5-80
bind 0.0.0.0:80
default_backend api_http
mode http
http-request redirect scheme https
backend api_http
mode http
frontend anbox-stream-gateway-lb-5-443
bind 0.0.0.0:443 ssl crt /var/lib/haproxy/default.pem no-sslv3
default_backend app-anbox-stream-gateway
mode http
backend app-anbox-stream-gateway
mode http
balance leastconn
server anbox-stream-gateway-0-4000 10.212.218.61:4000 check ssl verify
none inter 2000 rise 2 fall 5 maxconn 4096
server anbox-stream-gateway-1-4000 10.212.218.93:4000 check ssl verify
none inter 2000 rise 2 fall 5 maxconn 4096
server anbox-stream-gateway-2-4000 10.212.218.144:4000 check ssl verify
none inter 2000 rise 2 fall 5 maxconn 4096
------------------------------------
The crash occurs after a first few HTTP requests going through and
happens again when systemd restarts the service.
The bug is already reported in Debian https://bugs.debian.org/cgi-
bin/bugreport.cgi?bug=921981 and upstream at
https://github.com/haproxy/haproxy/issues/40
Using the 1.8.19-1+deb10u2 package from Debian fixes the crash.
To manage notifications about this bug go to:
https://bugs.launchpad.net/haproxy/+bug/1884149/+subscriptions
_______________________________________________
Mailing list: https://launchpad.net/~ubuntu-ha
Post to : [email protected]
Unsubscribe : https://launchpad.net/~ubuntu-ha
More help : https://help.launchpad.net/ListHelp
