Re: Heath check responds up even when server is down
Hi, On Fri, 2020-10-30 at 00:49 +, Wesley Lukehart wrote: > To recap; > Exchange says component is Inactive > IIS is up and still serving content > healthcheck.htm page does not load, is down, unavailable, what have you > haproxy gets 200 response from health check that supposedly isn’t > available Have you tested with curl / wget from haproxy server if IIS/Exchange returns stautus=200 for /oab/healthcheck.htm ? curl -v -k https://ip.addr.e.ss/oab/healthcheck.htm and # this probably sends "correct" iis.exchange.domain.com SNI to iis server, # maybe iis/exchange needs SNI to serve correct file/status ? curl -v -k --resolve iis.exchange.domain.com:443:iis.ip.here https://iis.exchange.domain.com/oab/healthcheck.htm > Here are relevant haproxy logs showing the health check as good and > content still being proxied, even though the component is inactive (ie > health check page is not accessible) > Oct 29 14:51:39 haproxy: [WARNING] 302/145139 (93952) : > Health check for server be_ex2019_oab/ succeeded, reason: > Layer7 check passed, code: 200, info: "HTTP status check returned code > <3C>200<3E>", check duration: 8ms, status: 3/3 UP. > Looking at the IIS logs, when the component is active, I see the GET > requests from my workstations IP. When the component is inactive, no GET > request is logged from my workstation. > In addition, weather the service is active or inactive, IIS logs GET > requests from the haproxy servers: > 2020-10-30 00:13:01 10.168.99.91 GET /oab/healthcheck.htm - 443 - > - - 200 0 0 1 > 2020-10-30 00:13:11 10.168.99.91 GET /oab/healthcheck.htm - 443 - > - - 200 0 0 1 So both haproxy and IIS log show that /oab/healthcheck.htm is served with status=200 to haproxy ? When you test /oab/healthcheck.htm with browser what url do you use: https://correct.domain.com/oab/healthcheck.htm or https://ip.addr.es.s/oab/healthcheck.htm ? Do you get different result with ip or hostname ? -Jarno -- Jarno Huuskonen
RE: Heath check responds up even when server is down
r implementation. So I don't think the issue I am having is attributable to any particular version of haproxy. Does anyone have any ideas on how I can get haproxy to recognize when Exchange is in maintenance mode? Its baffling to me how the health check can pass and get a 200 response when the page isn't accessible by any other means (browsers, wget). It seems haproxy's health check requests are fundamentally different than a regular browser request and that this what is driving the difference in response. Hopefully there is a way to configure haproxy to request a health check in such a way that an Exchange service in maintenance mode is properly detected as down. Thanks, -Luke From: Wesley Lukehart Sent: Wednesday, October 14, 2020 18:28 To: 'haproxy@formilux.org' Subject: Heath check responds up even when server is down Hello fine people. Short time lurker, first time poster. Was on version 2.0.5 with CentOS 7.6 and everything was working fine with Exchange 2019. Upgraded to 2.2.3 and now when we put Exchange into maintenance mode HAProxy does not change status - it reports that all services are still up (L7OK/200). Example backend: backend be_ex2019_oab mode http balance roundrobin option httpchk GET /oab/healthcheck.htm option log-health-checks http-check expect status 200 server :443 check ssl inter 15s verify required ca-file server :443 check ssl inter 15s verify required ca-file If I stop the app pool for a service in IIS, or stop all of IIS, HAProxy will properly show the service/services as down - as it gets a non 200 response (503 or 404). When putting the Exchange server into maintenance mode, there is no http response. When I check with a browser I get "ERR_HTTP2_PROTOCOL_ERROR" or "Secure Connection Failed". Basically no response. When I check with wget from the haproxy server I get "HTTP request sent, awaiting response... Read error (Connection reset by peer) in headers." Yet HAProxy is happy and continues to try to send mail to the down server - not good. Any Ideas? I just tried 2.2.4 and no joy. Thanks, -Luke
RE: Heath check responds up even when server is down
Correct - nothing in the logs that show L7TOUT/L7RSP. I actually change the check interval to 2s when testing so I don't have to wait as long when testing. -Original Message- From: Christopher Faulet Sent: Thursday, October 15, 2020 09:02 To: Wesley Lukehart ; haproxy@formilux.org Subject: Re: Heath check responds up even when server is down Le 15/10/2020 à 03:27, Wesley Lukehart a écrit : > Hello fine people. Short time lurker, first time poster. > > Was on version 2.0.5 with CentOS 7.6 and everything was working fine > with Exchange 2019. > > Upgraded to 2.2.3 and now when we put Exchange into maintenance mode > HAProxy does not change status - it reports that all services are still up > (L7OK/200). > > Example backend: > > backend be_ex2019_oab > > mode http > > balance roundrobin > > option httpchk GET /oab/healthcheck.htm > > option log-health-checks > > http-check expect status 200 > > server :443 check ssl inter 15s verify > required ca-file > > server :443 check ssl inter 15s verify > required ca-file > > If I stop the app pool for a service in IIS, or stop all of IIS, > HAProxy will properly show the service/services as down - as it gets a > non 200 response (503 or 404). > > When putting the Exchange server into maintenance mode, there is no http > response. > > When I check with a browser I get "ERR_HTTP2_PROTOCOL_ERROR" or > "Secure Connection Failed". Basically no response. > > When I check with wget from the haproxy server I get "HTTP request > sent, awaiting response... Read error (Connection reset by peer) in headers." > > Yet HAProxy is happy and continues to try to send mail to the down > server - not good. > > Any Ideas? > > I just tried 2.2.4 and no joy. > Hi, Just to be sure, when you says HAProxy still see the server up, there is no Health check errors in your logs ? No L7TOUT/L7RSP ? Because with your configuration and a default "fall" server parameter (3), you should wait at least 45s (3 x 15s) to see the server down. -- Christopher Faulet
RE: Heath check responds up even when server is down
Thanks for the suggestion. I tried this and there was no change in behavior. -Original Message- From: Jarno Huuskonen Sent: Thursday, October 15, 2020 00:25 To: Wesley Lukehart ; haproxy@formilux.org Subject: Re: Heath check responds up even when server is down Hi, On Thu, 2020-10-15 at 01:27 +, Wesley Lukehart wrote: > Hello fine people. Short time lurker, first time poster. > > Was on version 2.0.5 with CentOS 7.6 and everything was working fine > with Exchange 2019. > Upgraded to 2.2.3 and now when we put Exchange into maintenance mode > HAProxy does not change status – it reports that all services are > still up (L7OK/200). > > Example backend: > backend be_ex2019_oab > mode http > balance roundrobin > option httpchk GET /oab/healthcheck.htm > option log-health-checks > http-check expect status 200 > server :443 check ssl inter 15s verify > required ca-file > server :443 check ssl inter 15s verify > required ca-file > > If I stop the app pool for a service in IIS, or stop all of IIS, > HAProxy will properly show the service/services as down – as it gets a > non 200 response (503 or 404). > > When putting the Exchange server into maintenance mode, there is no > http response. > When I check with a browser I get “ERR_HTTP2_PROTOCOL_ERROR” or > “Secure Connection Failed”. Basically no response. > When I check with wget from the haproxy server I get “HTTP request > sent, awaiting response... Read error (Connection reset by peer) in headers.” > Yet HAProxy is happy and continues to try to send mail to the down > server – not good. > > Any Ideas? Does the health check work if you try with something like this: option httpchk http-check connect ssl http-check send meth GET uri /oab/healthcheck.htm ver HTTP/1.1 hdr Host somehost.example.org http-check expect status 200 ( https://cbonte.github.io/haproxy-dconv/2.2/configuration.html#4.2-http-check%20connect ) -Jarno -- Jarno Huuskonen
Re: Heath check responds up even when server is down
Le 15/10/2020 à 03:27, Wesley Lukehart a écrit : Hello fine people. Short time lurker, first time poster. Was on version 2.0.5 with CentOS 7.6 and everything was working fine with Exchange 2019. Upgraded to 2.2.3 and now when we put Exchange into maintenance mode HAProxy does not change status – it reports that all services are still up (L7OK/200). Example backend: backend be_ex2019_oab mode http balance roundrobin option httpchk GET /oab/healthcheck.htm option log-health-checks http-check expect status 200 server :443 check ssl inter 15s verify required ca-file server :443 check ssl inter 15s verify required ca-file If I stop the app pool for a service in IIS, or stop all of IIS, HAProxy will properly show the service/services as down – as it gets a non 200 response (503 or 404). When putting the Exchange server into maintenance mode, there is no http response. When I check with a browser I get “ERR_HTTP2_PROTOCOL_ERROR” or “Secure Connection Failed”. Basically no response. When I check with wget from the haproxy server I get “HTTP request sent, awaiting response... Read error (Connection reset by peer) in headers.” Yet HAProxy is happy and continues to try to send mail to the down server – not good. Any Ideas? I just tried 2.2.4 and no joy. Hi, Just to be sure, when you says HAProxy still see the server up, there is no Health check errors in your logs ? No L7TOUT/L7RSP ? Because with your configuration and a default "fall" server parameter (3), you should wait at least 45s (3 x 15s) to see the server down. -- Christopher Faulet
Re: Heath check responds up even when server is down
Hi, On Thu, 2020-10-15 at 01:27 +, Wesley Lukehart wrote: > Hello fine people. Short time lurker, first time poster. > > Was on version 2.0.5 with CentOS 7.6 and everything was working fine with > Exchange 2019. > Upgraded to 2.2.3 and now when we put Exchange into maintenance mode > HAProxy does not change status – it reports that all services are still up > (L7OK/200). > > Example backend: > backend be_ex2019_oab > mode http > balance roundrobin > option httpchk GET /oab/healthcheck.htm > option log-health-checks > http-check expect status 200 > server :443 check ssl inter 15s verify required > ca-file > server :443 check ssl inter 15s verify required > ca-file > > If I stop the app pool for a service in IIS, or stop all of IIS, HAProxy > will properly show the service/services as down – as it gets a non 200 > response (503 or 404). > > When putting the Exchange server into maintenance mode, there is no http > response. > When I check with a browser I get “ERR_HTTP2_PROTOCOL_ERROR” or “Secure > Connection Failed”. Basically no response. > When I check with wget from the haproxy server I get “HTTP request sent, > awaiting response... Read error (Connection reset by peer) in headers.” > Yet HAProxy is happy and continues to try to send mail to the down server > – not good. > > Any Ideas? Does the health check work if you try with something like this: option httpchk http-check connect ssl http-check send meth GET uri /oab/healthcheck.htm ver HTTP/1.1 hdr Host somehost.example.org http-check expect status 200 ( https://cbonte.github.io/haproxy-dconv/2.2/configuration.html#4.2-http-check%20connect ) -Jarno -- Jarno Huuskonen
Heath check responds up even when server is down
Hello fine people. Short time lurker, first time poster. Was on version 2.0.5 with CentOS 7.6 and everything was working fine with Exchange 2019. Upgraded to 2.2.3 and now when we put Exchange into maintenance mode HAProxy does not change status - it reports that all services are still up (L7OK/200). Example backend: backend be_ex2019_oab mode http balance roundrobin option httpchk GET /oab/healthcheck.htm option log-health-checks http-check expect status 200 server :443 check ssl inter 15s verify required ca-file server :443 check ssl inter 15s verify required ca-file If I stop the app pool for a service in IIS, or stop all of IIS, HAProxy will properly show the service/services as down - as it gets a non 200 response (503 or 404). When putting the Exchange server into maintenance mode, there is no http response. When I check with a browser I get "ERR_HTTP2_PROTOCOL_ERROR" or "Secure Connection Failed". Basically no response. When I check with wget from the haproxy server I get "HTTP request sent, awaiting response... Read error (Connection reset by peer) in headers." Yet HAProxy is happy and continues to try to send mail to the down server - not good. Any Ideas? I just tried 2.2.4 and no joy. Thanks, -Luke
