Hi all,
Looking for some help with a networking issue we've been debugging for
several days. We use haproxy to TCP load-balance between Kafka Connectors
and a Percona MySQL cluster. In this set-up, the connectors (i.e., Java
JDBC) maintain long-running connections and read the database binlogs. This
includes regular polling.
We have seen connection issues starting with haproxy 2.4 and persisting
through 2.8 which result in the following errors in MySQL:
2023-07-31T17:25:45.745607Z 3364649 [Note] Got an error reading
communication packets
As you can see, this doesn't include a host or user and is happening early
in the connection. The host cache shows handshake errors here regularly
accumulating.
We were unable to see errors on the haproxy side with tcplog on and have
been unable to get useful information from tcpdump, netstat, etc.
We are aware FE/BE connection closure behavior changed in 2.4. The 2.4
option of idle-close-on-response seemed like a possible solution but isn't
compatible with mode tcp, so we're not sure what's happening here or next
steps for debugging. Appreciate any help or guidance here.
We're running haproxy in Kubernetes using the official container, and are
also not seeing any issues with current haproxy versions with our other
(Python) applications.
A simplified version of our config:
global
daemon
maxconn 25000
defaults
balance roundrobin
option dontlognull
option redispatch
timeout http-request 5s
timeout queue 1m
timeout connect 4s
timeout client 50s
timeout server 30s
timeout http-keep-alive 10s
timeout check 10s
retries 3
frontend main_writer
bind :3306
mode tcp
timeout client 30s
timeout client-fin 30s
default_backend main_writer
backend main_writer
mode tcp
balance leastconn
option httpchk
timeout server 30s
timeout tunnel 3h
server db1 <ip>:3306 check port 9200 on-marked-down shutdown-sessions
weight 100 inter 3s rise 1 fall 2
server db2 <ip>:3306 check port 9200 on-marked-down shutdown-sessions
weight 100 backup
server db3 <ip>:3306 check port 9200 on-marked-down shutdown-sessions
weight 100 backup
*David GreenwaldSenior Site Reliability
[email protected] <[email protected]>*
--
The contents of this communication are confidential. If you are not the
intended recipient, please immediately notify the sender by reply email and
delete this message and its attachments, if any.
