Thanks for the response! That seems unlikely, we're doing an httpchk to the clustercheck utility <https://docs.percona.com/percona-xtradb-cluster/5.7/howtos/virt_sandbox.html> following the pxc reference architecture, so not actually making a direct database request from haproxy. We're also accessing the database with the same healthchecks from Python web applications without any issues, we're just seeing this from the long-lived connections, specifically JDBC sink connectors and Debezium as a source connector.
This seems to be a pretty esoteric issue and we've come up empty in our Googling, unfortunately. *David GreenwaldSenior Site Reliability [email protected] <[email protected]>* On Tue, Aug 1, 2023 at 8:25 PM Willy Tarreau <[email protected]> wrote: > Hi David, > > On Tue, Aug 01, 2023 at 05:11:48PM -0700, David Greenwald wrote: > > Hi all, > > > > Looking for some help with a networking issue we've been debugging for > > several days. We use haproxy to TCP load-balance between Kafka Connectors > > and a Percona MySQL cluster. In this set-up, the connectors (i.e., Java > > JDBC) maintain long-running connections and read the database binlogs. > This > > includes regular polling. > > > > We have seen connection issues starting with haproxy 2.4 and persisting > > through 2.8 which result in the following errors in MySQL: > > > > 2023-07-31T17:25:45.745607Z 3364649 [Note] Got an error reading > > communication packets > > > > As you can see, this doesn't include a host or user and is happening > early > > in the connection. The host cache shows handshake errors here regularly > > accumulating. > > > > We were unable to see errors on the haproxy side with tcplog on and have > > been unable to get useful information from tcpdump, netstat, etc. > > > > We are aware FE/BE connection closure behavior changed in 2.4. The 2.4 > > option of idle-close-on-response seemed like a possible solution but > isn't > > compatible with mode tcp, so we're not sure what's happening here or next > > steps for debugging. Appreciate any help or guidance here. > > The changes you're referring to are indeed only in HTTP mode. Have you > tried without health checks ? I'm indeed wondering if the handshake errors > you're observing are not simply health checks, which would explain why > you're not seeing them in your traffic logs. > > Willy > -- The contents of this communication are confidential. If you are not the intended recipient, please immediately notify the sender by reply email and delete this message and its attachments, if any.
