Re: PH disconnects, but "show errors" has 0 entries ?
Le 10/19/21 à 16:49, Jim Freeman a écrit : OK - this is weird (so don't shoot the messenger?). With more tcpdump-ing and examination, the back-end service logs that it sent a response, but 1) tcpdump running on the haproxy instance never sees the response ! a) 2 proxies - an AWS ELB and on-instance nginx - lie between HAProxy instance and the service 2) sans any response (and within 0.2 to 13 seconds of the request send), HAProxy initiates the PH/500 to the client! It would make sense to me if any timeouts or disconnects were involved - HAProxy would report an [sS][DH] or somesuch. And reverting the sending of the "content-security-policy: frame-ancestors ..." and "x-frame-options: ..." response(!) headers makes the problem disappear again. You'll rightly point out that HTTP/1.1 is stateless, and that the prior history of the request/response stream (and response headers sent to the client) shouldn't affect the (non-)response to a given request. Any clues as to how/why the PH/500 might be generated without a response to trigger it would be most eagerly received. While it is entirely likely this will wind up being a "nut loose on the keyboard" issue, I just thought I'd share my observations and befuddlement ... First of all, I missed a point. The 2.2.8 is quite old. You must upgrade first. Then, have you check the rewrite error counters on your backend ? Because, AFAIK, it is the only place where a 500 is possible with the PH termination state. If you are using http-after-response rules, it may explain this error. However, share your redacted configuration too. It can help me to explain what you observe. -- Christopher Faulet
Re: PH disconnects, but "show errors" has 0 entries ?
OK - this is weird (so don't shoot the messenger?). With more tcpdump-ing and examination, the back-end service logs that it sent a response, but 1) tcpdump running on the haproxy instance never sees the response ! a) 2 proxies - an AWS ELB and on-instance nginx - lie between HAProxy instance and the service 2) sans any response (and within 0.2 to 13 seconds of the request send), HAProxy initiates the PH/500 to the client! It would make sense to me if any timeouts or disconnects were involved - HAProxy would report an [sS][DH] or somesuch. And reverting the sending of the "content-security-policy: frame-ancestors ..." and "x-frame-options: ..." response(!) headers makes the problem disappear again. You'll rightly point out that HTTP/1.1 is stateless, and that the prior history of the request/response stream (and response headers sent to the client) shouldn't affect the (non-)response to a given request. Any clues as to how/why the PH/500 might be generated without a response to trigger it would be most eagerly received. While it is entirely likely this will wind up being a "nut loose on the keyboard" issue, I just thought I'd share my observations and befuddlement ... https://www.mail-archive.com/haproxy@formilux.org/msg41308.html "This computer stuff is hard ..." On Tue, Oct 19, 2021 at 3:24 AM Christopher Faulet wrote: > Le 10/13/21 à 8:30 PM, Jim Freeman a écrit : > > In adding a couple of new security response headers via haproxy.cfg (one > is 112 > > bytes, the other 32), a few requests are now getting 500 status (PH > session > > state) responses, but "show errors" has 0 entries? Most responses > succeed (all > > have the additional headers), so it's not a problem with the new headers > themselves. > > > > If haproxy generates a PH/500, shouldn't "show errors" show details of > the > > offending response ? > > > > Thanks, > > ...jfree > > == > > # echo "show info" | socat stdio /run/haproxy/stats.sock | grep ^Version: > > Version: 2.2.8-1~bpo10+1 > > > > # echo "show errors -1" | socat - /run/haproxy/stats.sock > > Total events captured on [13/Oct/2021:18:24:15.819] : 0 > > > > # cat /etc/debian_version > > 10.11 > > Hi, > > Only parsing errors are reported by "show errors" command. Here PH/500 > error is > most probably due to a header rewrite error. I have not deeply checked > however. > You can verify my assumption by checking the "wrew" counter in "show > stats" > command output on the stats socket. > > Header rewrite errors are triggered when there is not enough space in the > buffer > to perform the rewrites. By default, 1024 Bytes are reserved in the > buffer, to > be sure to have enough space to perform some rewrites. If you add many > headers > in the response, it may be the problem. You can increase the reserve by > setting > "tune.maxrewrite" global parameter. > > When such error is encountered, HAProxy returns a 500-Internal-Error > response. > You can change that to make it fails silently. To do so, take a look at > the > "strict-mode" http-response action. > > -- > Christopher Faulet >
Re: PH disconnects, but "show errors" has 0 entries ?
Many thanks for your insight and response - I'll check that out. On Tue, Oct 19, 2021 at 3:24 AM Christopher Faulet wrote: > Le 10/13/21 à 8:30 PM, Jim Freeman a écrit : > > In adding a couple of new security response headers via haproxy.cfg (one > is 112 > > bytes, the other 32), a few requests are now getting 500 status (PH > session > > state) responses, but "show errors" has 0 entries? Most responses > succeed (all > > have the additional headers), so it's not a problem with the new headers > themselves. > > > > If haproxy generates a PH/500, shouldn't "show errors" show details of > the > > offending response ? > > > > Thanks, > > ...jfree > > == > > # echo "show info" | socat stdio /run/haproxy/stats.sock | grep ^Version: > > Version: 2.2.8-1~bpo10+1 > > > > # echo "show errors -1" | socat - /run/haproxy/stats.sock > > Total events captured on [13/Oct/2021:18:24:15.819] : 0 > > > > # cat /etc/debian_version > > 10.11 > > Hi, > > Only parsing errors are reported by "show errors" command. Here PH/500 > error is > most probably due to a header rewrite error. I have not deeply checked > however. > You can verify my assumption by checking the "wrew" counter in "show > stats" > command output on the stats socket. > > Header rewrite errors are triggered when there is not enough space in the > buffer > to perform the rewrites. By default, 1024 Bytes are reserved in the > buffer, to > be sure to have enough space to perform some rewrites. If you add many > headers > in the response, it may be the problem. You can increase the reserve by > setting > "tune.maxrewrite" global parameter. > > When such error is encountered, HAProxy returns a 500-Internal-Error > response. > You can change that to make it fails silently. To do so, take a look at > the > "strict-mode" http-response action. > > -- > Christopher Faulet >
Re: PH disconnects, but "show errors" has 0 entries ?
Le 10/13/21 à 8:30 PM, Jim Freeman a écrit : In adding a couple of new security response headers via haproxy.cfg (one is 112 bytes, the other 32), a few requests are now getting 500 status (PH session state) responses, but "show errors" has 0 entries? Most responses succeed (all have the additional headers), so it's not a problem with the new headers themselves. If haproxy generates a PH/500, shouldn't "show errors" show details of the offending response ? Thanks, ...jfree == # echo "show info" | socat stdio /run/haproxy/stats.sock | grep ^Version: Version: 2.2.8-1~bpo10+1 # echo "show errors -1" | socat - /run/haproxy/stats.sock Total events captured on [13/Oct/2021:18:24:15.819] : 0 # cat /etc/debian_version 10.11 Hi, Only parsing errors are reported by "show errors" command. Here PH/500 error is most probably due to a header rewrite error. I have not deeply checked however. You can verify my assumption by checking the "wrew" counter in "show stats" command output on the stats socket. Header rewrite errors are triggered when there is not enough space in the buffer to perform the rewrites. By default, 1024 Bytes are reserved in the buffer, to be sure to have enough space to perform some rewrites. If you add many headers in the response, it may be the problem. You can increase the reserve by setting "tune.maxrewrite" global parameter. When such error is encountered, HAProxy returns a 500-Internal-Error response. You can change that to make it fails silently. To do so, take a look at the "strict-mode" http-response action. -- Christopher Faulet
Re: PH disconnects, but "show errors" has 0 entries ?
Nope - never mind. Plenty of successful traffic with the sec-ch-ua* headers. I'll keep poking re: PH/500 w/o "show errors", and confess here when/how I find it is the result of being ignernt. On Mon, Oct 18, 2021 at 11:41 AM Jim Freeman wrote: > Ran tcpdump on the proxy in search of useful detail. > Saw these unfamiliar (to me) headers on the PH/500 'd request : > > sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="90" > sec-ch-ua-mobile: ?0 > > Googled, found : https://www.chromium.org/updates/ua-ch, was a tad > FUD'd by > === > Possible Site Compatibility Issue > UA-CH is an additive feature, which adds two new request headers that are > sent by default: `sec-ch-ua` and `sec-ch-ua-mobile`. Those request headers > are based off of Structured Field Values, an emerging standard related to > HTTP header values. They contain characters that, though permitted in the > HTTP specification, weren’t previously common in request headers, such as > double-quotes (“), equal signs (=), forward-slashes (/), and question marks > (?). Some Web-Application-Firewall (WAF) software, as well as backend > security measures, may mis-categorize those new characters as “suspicious”, > and as such, block those requests. > === > > HAProxy tends to be up on all such things, but any chance the PH/500 could > be related ? > > Thanks, > ...jfree > https://www.mail-archive.com/haproxy@formilux.org/msg41272.html Added headers : content-security-policy: frame-ancestors 'self' https://*.primarydomain.org https://*.related.domain.org; x-frame-options: SAMEORIGIN
Re: PH disconnects, but "show errors" has 0 entries ?
Ran tcpdump on the proxy in search of useful detail. Saw these unfamiliar (to me) headers on the PH/500 'd request : sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="90" sec-ch-ua-mobile: ?0 Googled, found : https://www.chromium.org/updates/ua-ch, was a tad FUD'd by === Possible Site Compatibility Issue UA-CH is an additive feature, which adds two new request headers that are sent by default: `sec-ch-ua` and `sec-ch-ua-mobile`. Those request headers are based off of Structured Field Values, an emerging standard related to HTTP header values. They contain characters that, though permitted in the HTTP specification, weren’t previously common in request headers, such as double-quotes (“), equal signs (=), forward-slashes (/), and question marks (?). Some Web-Application-Firewall (WAF) software, as well as backend security measures, may mis-categorize those new characters as “suspicious”, and as such, block those requests. === HAProxy tends to be up on all such things, but any chance the PH/500 could be related ? Thanks, ...jfree
PH disconnects, but "show errors" has 0 entries ?
In adding a couple of new security response headers via haproxy.cfg (one is 112 bytes, the other 32), a few requests are now getting 500 status (PH session state) responses, but "show errors" has 0 entries? Most responses succeed (all have the additional headers), so it's not a problem with the new headers themselves. If haproxy generates a PH/500, shouldn't "show errors" show details of the offending response ? Thanks, ...jfree == # echo "show info" | socat stdio /run/haproxy/stats.sock | grep ^Version: Version: 2.2.8-1~bpo10+1 # echo "show errors -1" | socat - /run/haproxy/stats.sock Total events captured on [13/Oct/2021:18:24:15.819] : 0 # cat /etc/debian_version 10.11
