Thank you Joao for starting this thread, and Moemen for your reply.
More information:
1) Something on the server side is sending RST packets to client initiated
connections.
2) When it happens, all connections from a single IP address are reaped.
However, it doesn't always happen to the same IP address.
3) Sketch architecture: The server is running HAProxy on port 443. HAProxy
then sends the HTTP connections to various backends. As Joao indicated,
this is a Kubernetes Ingress setup, but the above is the TL;DR.
4) In my analysis of the logs, I can't find evidence of HAProxy doing the
reaping, but it might be my lack of familiarity.
5) Here is our log-format. This facilitates JSON parsing of logs.:
log-format
{\"type\":\"haproxy-ingress\",\"protocol\":\"http\",\"timestamp\":\"%t\",\"client_ip\":%{+Q}ci,\"client_port\":%{+Q}cp,\"frontend_name\":%{+Q}ft,\"backend_name\":%{+Q}b,\"backend_ip\":%{+Q}s,\"time_handshake\":%{+Q}Th,\"time_idle_before_request\":%{+Q}Ti,\"time_receive_full_request\":%{+Q}TR,\"time_wait_send_backend\":%{+Q}Tw,\"time_backend_ack\":%{+Q}Tc,\"time_backend_processing\":%{+Q}Tr,\"time_delay_send_to_client\":%{+Q}Td,\"waittime_on_client\":%{+Q}Tq,\"active_request_time\":%{+Q}Ta,\"total_request_time\":%{+Q}Tt,\"http_status_code\":%{+Q}ST,\"client_bytes_uploaded\":%{+Q}U,\"bytes_sent\":%{+Q}B,\"termination_status\":\"%tsc\",\"active_conns\":%{+Q}ac,\"frontend_conns\":%{+Q}fc,\"backend_conns\":%{+Q}bc,\"server_conns\":%{+Q}sc,\"retry_conns\":%{+Q}rc,\"server_queue\":%{+Q}sq,\"backend_queue\":%{+Q}bq,\"request\":%{+Q,+E}r}
6) Around the 15:15 minute in the logs, there were 132 RSTs. They were all
within one second.
7) The client IPs have been changed, and the domains have been changed.
Otherwise the logs are unchanged. The IP changes are all 1:1. They
include the 15:13-15:16 minutes.
8) The logs can be found here:
https://gist.github.com/zapman449/ab043d9849ed78826bdb0f343559a230
Thank you for any insight;
Jason
