Hi,
I am seeing quite strange and not simple to reproduce behavior found
during stress testing of https://github.com/codingfuture/puppet-cfdb .
After a long testing with different options, I am quite sure the issue
comes from epoll support as "noepoll" fixes the problem.
Here's the scenario:
1. There are several DB clusters accessible through TCP
2. HAProxy 1.6.5 on Debian Jessie accepts clients on UNIX sockets
under /run/{service}/ (easier auto-configuration, security and per
role maxconn control)
3. external-check python-based (yep..) scripts are used to properly
identify ready nodes in clusters
4. As cluster connections may be wrapped in TLS, external-check
scripts use dedicated no-check "listen" blocks in HAProxy, but do not
connect directly.
5. As there is a known recently discovered issue with stdout/stderr
data clobbering, I made sure external-check scripts to close mentioned
descriptors (the example below uses /dev/null instead).
What is seen when epoll is enabled (splice is disabled):
- both UNIX and TCP connections fail in the middle of transmission
What is seen with "noepoll":
- both UNIX and TCP connections work without issues
What is seen when epoll is enabled, but external-check is disabled:
- both UNIX and TCP connections work without issues
So, I have a clue that SIGCHILD with followed EINTR on epoll leads to
UNIX connections getting dropped. However, I have not tried to dig
into code yet.
Below are steps to reproduce:
1. /tmp/haproxy.conf:
-----------------------
global
external-check
#noepoll
nosplice
defaults
timeout connect 5s
timeout client 10m
timeout server 10m
timeout check 2s
option abortonclose
option dontlog-normal
option dontlognull
option srvtcpka
option clitcpka
backend mysql:myclust1
mode tcp
option external-check
retries 1
balance first
external-check command /tmp/test_check.sh
# Note: 10ms - is to reproduce the problem
server dbclust1_example_com_3306 10.10.2.20:3306 check fall 2 rise
1 inter 10ms fastinter 10ms weight 100
server dbclust2_example_com_3306 10.10.2.21:3306 check fall 2 rise
1 inter 10ms fastinter 10ms weight 100 backup
listen check__dbclust1_example_com_3306
mode tcp
retries 0
maxconn 6
# /tmp is used only to reproduce, not for security
bind unix@/tmp/check__dbclust1_example_com_3306.sock user
cfhaproxy group cfhaproxy mode 660
# just for testing purposes
bind 127.0.0.1:3306
server check__dbclust1_example_com_3306 10.10.2.20:3306
-----------------------
2. /tmp/test_check.sh:
-----------------------
#!/bin/sh
/usr/bin/mysql --socket=/tmp/check__dbclust1_example_com_3306.sock -e
'SELECT 1;' >/dev/null 2>&1
-----------------------
3. Setup simple MySQL instance, protected by password
-----------------------
# OS-specific way
4. Start a test instance
-----------------------
strace haproxy -d -- /tmp/haproxy.conf
5. Run test with either TCP or UNIX socket
-----------------------
for i in $(seq 1 1000); do echo -n "$i: "; mysql --host 127.0.0.1
--port 3306 -e 'SELECT 1;' --ssl-mode=DISABLED; done 2>&1 | grep -v
"Access denied for user"
or
for i in $(seq 1 1000); do echo -n "$i: "; mysql
--socket=/tmp/check__dbclust1_example_com_3306.sock -e 'SELECT 1;'
done 2>&1 | grep -v "Access denied for user"
3: ERROR 2013 (HY000): Lost connection to MySQL server at 'reading
authorization packet', system error: 0
5: ERROR 2013 (HY000): Lost connection to MySQL server at 'reading
authorization packet', system error: 0
7: ERROR 2013 (HY000): Lost connection to MySQL server at 'reading
authorization packet', system error: 0
12: ERROR 2013 (HY000): Lost connection to MySQL server at 'reading
authorization packet', system error: 0
13: ERROR 2013 (HY000): Lost connection to MySQL server at 'reading
authorization packet', system error: 0
17: ERROR 2013 (HY000): Lost connection to MySQL server at 'reading
authorization packet', system error: 0
... and so on ...
6. Run successful test with noepoll
-----------------------
Uncomment "noepoll".
Repeat test.
Observe no errors.
Thanks,
Andrey Galkin
