https://bz.apache.org/bugzilla/show_bug.cgi?id=63795
Bug ID: 63795
Summary: mod_proxy_balancer: disabling balancer member often
results in "AH01114: HTTP: failed to make connection
to backend"
Product: Apache httpd-2
Version: 2.4.25
Hardware: PC
OS: Linux
Status: NEW
Severity: normal
Priority: P2
Component: mod_proxy_balancer
Assignee: bugs@httpd.apache.org
Reporter: jamessto...@codeweavers.net
Target Milestone: ---
~~ Bug Description ~~
In the situation whereby an HTTP balancer is setup using mod_proxy_balancer and
mod_proxy_http and proxy-initial-not-pooled is enabled, an disabling a balancer
member can on occasion put the balancer member in an errored state and result
in an "AH01114: HTTP: failed to make connection to backend" error written to
the error.log.
We have seen this issue in our production environment and successfully
reproduced it in our lab environment.
>From examining multiple traffic captures, it appears the following is
happening:
1) The balancer receives a HTTP request to be proxied
2) The balancer elects a balancer member and establishes a TCP connection with
the elected balancer member
3) Before the balancer proxies any HTTP traffic over the established TCP
connection, the balancer manager receives a request disable the elected
balancer member
4) The balancer closes the established TCP connection and puts the balancer
member into an error state rather than the expected disabled state, logging
"AH01114: HTTP: failed to make connection to backend"
5) The balancer elects another balancer member and proxies the HTTP request as
normal. The client making the request does not experience an error.
Our monitoring scrapes the HTTP balancer manager page each minute and alerts if
any of the balancer members are in an error state. We run a continuous delivery
pipeline and balancer members are disabled in the balancer whilst our software
is deployed to them. We have 4 Apache2 HTTP balancers and 12 balancer members.
Yesterday 71 deploys were executed on our pipeline, which means each balancer
member was disabled, allowed to drain, deployed to and re-enabled 71 times.
This resulted in 30 alerts from our monitoring for a balancer member being in
an error state.
We had not observed this issue prior to upgrading to Apache 2.4.
~~ Expected Behaviour ~~
mod_proxy_balancer should not put an elected balancer member into an errored
state when the established TCP connection is closed before the HTTP request is
proxied because the balancer manager has recieved a POST request to disable the
elected balancer member. In this situation mod_proxy_balancer should put the
elected balancer member in a disabled state.
~~ Steps to reproduce ~~
Set up an apache2 http balancer (full configuraton below) with
mod_proxy_balancer and mod_proxy_http. 4 balancer members each running apache2
server in stock Debian configuration with libapache2-mod-php installed. Each
member server is serving up a php page an embedded random sleep (0 - 2
seconds):
cat /var/www/html/index.php
<html>
<?php
sleep(rand(0, 2));
?>
<head>
<title>httpserver01.cw.lab says
hello</title>
</head>
<body>
<h1>This page lovingly served by httpserver01</h1>
<h2>You're welcome!</h2>
</body>
</html>
ab (Apache Bench) is continuously making 50 concurrent requests targeted at the
balancer:
#!/bin/bash
while : ; do
ab -c50 -n50000 http://wwwexample.cw.lab/ 2>&1 | grep -P
'(Failed requests|Non-2xx responses):\s+\d+' | grep -v 'Failed requests:
0' | tee -a fails
done
Continuously cycle through the balancer members, disabling them, waiting 5
seconds and then re-enabling them:
#!/bin/bash
while : ; do
nonce=$(curl -s
http://httpbalancer.cw.lab/viphttpbalancer/__balancer-manager | grep --colour
-oPm 1 '\w{8}-(?:\w{4}-){3}\w{12}') || ( echo "Failed to get nonce" ; exit 1)
for i in {1..4} ; do
echo Removing httpserver0$i from httpbalancer balancer
curl -sS -w '%{http_code}' -o /dev/null
'http://httpbalancer.cw.lab/viphttpbalancer/__balancer-manager' --data
'b=vip-http-balancer' --data 'w_lf=10' --data 'w_ls=0' --data 'w_wr' --data
'w_rr' --data 'w_status_I=0' --data 'w_status_N=0' --data 'w_status_D=1' --data
'w_status_H=0' --data 'w_status_S=0' --data "w=http://192.168.13.7$i"; --data
'b=vip-http-balancer' --data "nonce=$nonce"; echo
sleep 5
echo Re-adding httpserver0$i to httpbalancer balancer
curl -sS -w '%{http_code}' -o /dev/null
'http://httpbalancer.cw.lab/viphttpbalancer/__balancer-manager' --data
'b=vip-http-balancer' --data 'w_lf=10' --data 'w_ls=0' --data 'w_wr' --data
'w_rr' --data 'w_status_I=0' --data 'w_status_N=0' --data 'w_status_D=0' --data
'w_status_H=0' --data 'w_status_S=0' --data "w=http://192.168.13.7$i"; --data
'b=vip-http-balancer' --data "nonce=$nonce"; echo
sleep 1
done
done
~~ HTTP Balancer Environment/Configuration details ~~
Enabled MPM: worker
List of enabled modules:
proxy_balancer (enabled by unknown)
filter (enabled by maintainer script)
alias (enabled by maintainer script)
setenvif (enabled by maintainer script)
dir (enabled by maintainer script)
headers (enabled by unknown)
lbmethod_byrequests (enabled by unknown)
slotmem_shm (enabled by unknown)
status (enabled by maintainer script)
proxy (enabled by unknown)
proxy_html (enabled by unknown)
ssl (enabled by unknown)
access_compat (enabled by maintainer script)
deflate (enabled by maintainer script)
env (enabled by maintainer script)
proxy_http (enabled by unknown)
authn_file (enabled by maintainer script)
authz_core (enabled by maintainer script)
authn_core (enabled by maintainer script)
mime (enabled by maintainer script)
authz_user (enabled by maintainer script)
negotiation (enabled by maintainer script)
rewrite (enabled by unknown)
xml2enc (enabled by unknown)
mpm_worker (enabled by unknown)
socache_shmcb (enabled by unknown)
auth_basic (enabled by maintainer script)
authz_host (enabled by maintainer script)
List of enabled configurations:
cw-lab-all-hosts (enabled by unknown)
charset (enabled by maintainer script)
serve-cgi-bin (enabled by maintainer script)
localized-error-pages (enabled by maintainer script)
Package: apache2
Version: 2.4.25-3+deb9u7*
-- System Information:
Debian Release: 9.11
APT prefers oldstable-updates
APT policy: (500, 'oldstable-updates'), (500, 'oldstable')
Architecture: amd64 (x86_64)
Kernel: Linux 4.9.0-11-amd64 (SMP w/2 CPU cores)
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8),
LANGUAGE=en_GB:en (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
Versions of packages apache2 depends on:
ii apache2-bin 2.4.25-3+deb9u7
ii apache2-data 2.4.25-3+deb9u7
ii apache2-utils 2.4.25-3+deb9u7
ii dpkg 1.18.25
ii init-system-helpers 1.48
ii lsb-base 9.20161125
ii mime-support 3.60
ii perl 5.24.1-3+deb9u5
ii procps 2:3.3.12-3+deb9u1
Versions of packages apache2 recommends:
pn ssl-cert <none>
Versions of packages apache2 suggests:
pn apache2-doc <none>
pn apache2-suexec-pristine | apache2-suexec-custom <none>
pn www-browser <none>
Versions of packages apache2-bin depends on:
ii libapr1 1.5.2-5
ii libaprutil1 1.5.4-3
ii libaprutil1-dbd-sqlite3 1.5.4-3
ii libaprutil1-ldap 1.5.4-3
ii libc6 2.24-11+deb9u4
ii libldap-2.4-2 2.4.44+dfsg-5+deb9u3
ii liblua5.2-0 5.2.4-1.1+b2
ii libnghttp2-14 1.18.1-1+deb9u1
ii libpcre3 2:8.39-3
ii libssl1.0.2 1.0.2s-1~deb9u1
ii libxml2 2.9.4+dfsg1-2.2+deb9u2
ii perl 5.24.1-3+deb9u5
ii zlib1g 1:1.2.8.dfsg-5
Versions of packages apache2-bin suggests:
pn apache2-doc <none>
pn apache2-suexec-pristine | apache2-suexec-custom <none>
pn www-browser <none>
Versions of packages apache2 is related to:
ii apache2 2.4.25-3+deb9u7
ii apache2-bin 2.4.25-3+deb9u7
-- Configuration Files:
/etc/apache2/mods-available/mpm_worker.conf:
<IfModule mpm_worker_module>
StartServers 2
MinSpareThreads 25
MaxSpareThreads 75
ThreadLimit 64
ThreadsPerChild 25
MaxRequestWorkers 400
MaxConnectionsPerChild 0
</IfModule>
/etc/apache2/mods-available/cw_lab_all_hosts.conf:
ProxyPreserveHost On
SetEnv proxy-nokeepalive 1
SetEnv proxy-initial-not-pooled 1
SetEnv proxy-sendchunked 1
/etc/apache2/sites-available/viphttpbalancer.conf
<VirtualHost viphttpbalancer:80>
ServerAdmin infrastruct...@codeweavers.net
DocumentRoot /var/www
<Directory />
Options FollowSymLinks
AllowOverride None
</Directory>
<Directory /var/www/>
Options FollowSymLinks
AllowOverride None
Require all granted
</Directory>
<Location /__server-status>
SetHandler server-status
</Location>
<Location /__balancer-manager>
SetHandler balancer-manager
</Location>
<Location /__heartbeat>
SetHandler server-status
</Location>
#This is not inherited, so activate each individually
RewriteEngine On
#Don't proxy things that start with __ !!
RewriteRule ^/__(.*)$ /__$1 [L]
<Proxy balancer://vip-http-balancer>
BalancerMember http://192.168.13.71 loadfactor=10
BalancerMember http://192.168.13.72 loadfactor=10
BalancerMember http://192.168.13.73 loadfactor=10
BalancerMember http://192.168.13.74 loadfactor=10
ProxySet lbmethod=byrequests
</Proxy>
RewriteRule ^(.*)$ balancer://vip-http-balancer/$1 [P,QSA]
</VirtualHost>
/etc/apache2/sites-available/local_eth1.conf
<VirtualHost local_eth1:80>
ErrorLog ${APACHE_LOG_DIR}/local_eth1.redir.error.log
LogLevel warn
CustomLog ${APACHE_LOG_DIR}/local_eth1.redir.access.log combined
SetOutputFilter proxy-html;INFLATE;DEFLATE
ProxyPass /viphttpbalancer/ http://viphttpbalancer/
<Location /viphttpbalancer/>
ProxyHTMLURLMap /__balancer-manager
/viphttpbalancer/__balancer-manager R
</Location>
</VirtualHost>
/etc/apache2/sites-available/000-default.conf
<VirtualHost vip1:80>
ServerName wwwexample.cw.lab
ServerAlias wwwexample
ServerAlias vip1
ServerAdmin webmaster@localhost
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
RewriteEngine On
RewriteRule ^/__(.*)$ /__$1 [L]
RequestHeader set X-Forwarded-Proto "http"
RewriteCond %{HTTP_HOST} ^.*\.cw\.lab$ [NC]
RewriteRule ^(.*)$ http://viphttpbalancer/$1 [P,QSA]
</VirtualHost>
~~ Footnotes ~~
*2.4.25-3+deb9u8 is broken for our environment as per
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=941202#5 /
https://bz.apache.org/bugzilla/show_bug.cgi?id=63688
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org
For additional commands, e-mail: bugs-h...@httpd.apache.org
