Ok! Here is a bunch of info that might better assist with the issue:
Each of our clients has an HAProxy install that forwards requests for 80 and
443 to 1025 and 1026 respectively. These requests are forwarded over TCP using
proxy protocol to our HAP instances.
Our HAP instances then SSL term the request and forward them off to our backend
on port 80.
See attached diagram which should better explain the entire flow.
During an outage due to the SSL handshakes failing, I was running TCPDump so I
could look through and discover what was causing the failure, I was able to
discover that we are receiving connection resets on some SSL connections. We
then tested all the SSL certs from our client side to our side to verify that
there is not a mismatched cert. This test was completed with no issues.
Here is a connection reset packet I found in that TCP Dump
29525 158.09621710.1.4.119 54.239.21.251TCP 5438740 → 443 [RST] Seq=3533 Win=0
Len=0
Frame 29523: 54 bytes on wire (432 bits), 54 bytes captured (432 bits)
Encapsulation type: Ethernet (1)
Arrival Time: Mar 17, 2016 14:58:07.345840000 PDT
[Time shift for this packet: 0.000000000 seconds]
Epoch Time: 1458251887.345840000 seconds
[Time delta from previous captured frame: 0.000020000 seconds]
[Time delta from previous displayed frame: 0.021655000 seconds]
[Time since reference or first frame: 158.096184000 seconds]
Frame Number: 29523
Frame Length: 54 bytes (432 bits)
Capture Length: 54 bytes (432 bits)
[Frame is marked: False]
[Frame is ignored: False]
[Protocols in frame: eth:ethertype:ip:tcp]
[Coloring Rule Name: TCP RST]
[Coloring Rule String: tcp.flags.reset eq 1]
Ethernet II, Src: 12:8d:18:05:0f:91 (12:8d:18:05:0f:91), Dst: 1e:8f:a6:6b:52:58
(1e:8f:a6:6b:52:58)
Destination: 1e:8f:a6:6b:52:58 (1e:8f:a6:6b:52:58)
Address: 1e:8f:a6:6b:52:58 (1e:8f:a6:6b:52:58)
.... ..1. .... .... .... .... = LG bit: Locally administered address
(this is NOT the factory default)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
Source: 12:8d:18:05:0f:91 (12:8d:18:05:0f:91)
Address: 12:8d:18:05:0f:91 (12:8d:18:05:0f:91)
.... ..1. .... .... .... .... = LG bit: Locally administered address
(this is NOT the factory default)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: $SRC IP Dst: $DST IP
0100 .... = Version: 4
.... 0101 = Header Length: 20 bytes
Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
0000 00.. = Differentiated Services Codepoint: Default (0)
.... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport
(0)
Total Length: 40
Identification: 0x5f56 (24406)
Flags: 0x02 (Don't Fragment)
0... .... = Reserved bit: Not set
.1.. .... = Don't fragment: Set
..0. .... = More fragments: Not set
Fragment offset: 0
Time to live: 64
Protocol: TCP (6)
Header checksum: 0x8018 [validation disabled]
[Good: False]
[Bad: False]
Source: $SourceIP
Destination: $DestinationIP
[Source GeoIP: Unknown]
[Destination GeoIP: Unknown]
Transmission Control Protocol, Src Port: 38740 (38740), Dst Port: 443 (443),
Seq: 3533, Len: 0
Source Port: 38740
Destination Port: 443
[Stream index: 2799]
[TCP Segment Len: 0]
Sequence number: 3533 (relative sequence number)
Acknowledgment number: 0
Header Length: 20 bytes
Flags: 0x004 (RST)
000. .... .... = Reserved: Not set
...0 .... .... = Nonce: Not set
.... 0... .... = Congestion Window Reduced (CWR): Not set
.... .0.. .... = ECN-Echo: Not set
.... ..0. .... = Urgent: Not set
.... ...0 .... = Acknowledgment: Not set
.... .... 0... = Push: Not set
.... .... .1.. = Reset: Set
[Expert Info (Warn/Sequence): Connection reset (RST)]
[Connection reset (RST)]
[Severity level: Warn]
[Group: Sequence]
.... .... ..0. = Syn: Not set
.... .... ...0 = Fin: Not set
[TCP Flags: *********R**]
Window size value: 0
[Calculated window size: 0]
[Window size scaling factor: 128]
Checksum: 0x5c2f [validation disabled]
[Good Checksum: False]
[Bad Checksum: False]
Urgent pointer: 0
From: Igor Cicimov
<[email protected]<mailto:[email protected]>>
Date: Thursday, March 17, 2016 at 8:40 PM
To: Zachary Punches <[email protected]<mailto:[email protected]>>
Cc: Baptiste <[email protected]<mailto:[email protected]>>,
"[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Subject: Re: Help! HAProxy randomly failing health checks!
On Fri, Mar 18, 2016 at 1:38 PM, Igor Cicimov
<[email protected]<mailto:[email protected]>> wrote:
On Fri, Mar 18, 2016 at 12:04 PM, Zachary Punches
<[email protected]<mailto:[email protected]>> wrote:
Yeah port 1027 is used for health checks over SSL.
This HAP forwards requests off to our databases. The databases have a string in
a table that indicates that the HAP instance can move all the way through the
entire process before it lights as green.
Our health checks in route 53 are setup to ping 1027 as the SSL port
From: Igor Cicimov
<[email protected]<mailto:[email protected]>>
Date: Thursday, March 17, 2016 at 4:18 PM
To: Zachary Punches <[email protected]<mailto:[email protected]>>
Cc: Baptiste <[email protected]<mailto:[email protected]>>,
"[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Subject: Re: Help! HAProxy randomly failing health checks!
So is port 1027 used for health checks over SSL or not? I don't see any ssl
settings on that port.
I see. In that case, since you are doing ssl pass-through in http mode, you
need to add ssl to your server line:
backend server0 ## added to allow gs ssl meta tag verification
reqrep ^GET\ /.*\ (HTTP/.*) GET\ /GlobalSignVerification\ \1
server server0_http
server0.domain.com:80/GlobalSignVerification/<http://server0.domain.com:80/GlobalSignVerification/>
so it becomes:
server server0_http
server0.domain.com:80/GlobalSignVerification/<http://server0.domain.com:80/GlobalSignVerification/>
ssl
if I understood your intention correctly and server0 is ssl enabled db.
# Route traffic based on domain
use_backend gs_verify if gs_texthtml or gs_user_agent ## allow gs meta
tag verification
Where is this backend in your config? Which domain is used fore the health
checks?
