Re: Random 502's and instant 504's after upgrading

2019-07-19 Thread Christopher Faulet

Le 19/07/2019 à 09:36, Sander Klein a écrit :

The show errors:

---
Total events captured on [19/Jul/2019:08:34:25.093] : 31

[19/Jul/2019:08:34:23.405] backend cluster1-xx (#11): invalid response
frontend webservices (#18), server xxx (#2), event #30, src
x.x.x.x:63290
buffer starts at 0 (including 0 out), 16268 free,
len 116, wraps at 16384, error at position 0
H1 connection flags 0x, H1 stream flags 0x4094
H1 msg state MSG_RPBEFORE(8), H1 msg flags 0x1404
H1 chunk len 0 bytes, H1 body len 0 bytes :

0
{"metadata":{"pagination":{"total":0,"rows":25,"currentPage":1,"pages"
00070+ :0},"facets":[],"activeFacets":[]},"media":[]}


Thanks. So the problem seems to be the same than the issue #176 on github 
(https://github.com/haproxy/haproxy/issues/176). I pushed a fix.



---

I also did this request with wget to see what the response should be,
and it seems that this is the first part of the 297229 bytes long body.
The response headers are:

---
HTTP/1.1 200 OK
Server: nginx
Date: Fri, 19 Jul 2019 07:32:03 GMT
Content-Type: application/json; charset=UTF-8
Transfer-Encoding: chunked
Vary: Accept-Encoding
Vary: Accept-Encoding
Cache-Control: private, must-revalidate
ETag: "178c3f242b0151fe57e02f6e8817ce3a"
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: POST, GET, OPTIONS, PUT, PATCH, DELETE,
HEAD
Length: unspecified [application/json]
---

Maybe the 'Length: unspecified' has something to do with it.



No, this line is reported by wget because there is no "Content-Length" header.

So, as I said, I pushed a fix 
(https://github.com/haproxy/haproxy/commit/03627245). It was backported to 2.0. 
Could you check if it fixes your issue about 502 errors ?


For 504 errors, I have no idea for now.

--
Christopher Faulet



Re: Random 502's and instant 504's after upgrading

2019-07-19 Thread Sander Klein

Hi Lukas and Christopher,

I've combined the answer of your 2 mails.

On 2019-07-18 17:17, Lukas Tribus wrote:

Could be related to:
https://github.com/haproxy/haproxy/issues/176


Probably, but I'm not doing HTTP/1 and I have not found a request to 
reproduce it with. It happens at random.



Can you provide the "show errors" output from the admin cli for those
requests, and possible try one of the mentioned workarounds
(http-reuse never or http-server-close)?


The show errors:

---
Total events captured on [19/Jul/2019:08:34:25.093] : 31

[19/Jul/2019:08:34:23.405] backend cluster1-xx (#11): invalid response
  frontend webservices (#18), server xxx (#2), event #30, src 
x.x.x.x:63290

  buffer starts at 0 (including 0 out), 16268 free,
  len 116, wraps at 16384, error at position 0
  H1 connection flags 0x, H1 stream flags 0x4094
  H1 msg state MSG_RPBEFORE(8), H1 msg flags 0x1404
  H1 chunk len 0 bytes, H1 body len 0 bytes :

  0  
{"metadata":{"pagination":{"total":0,"rows":25,"currentPage":1,"pages"

  00070+ :0},"facets":[],"activeFacets":[]},"media":[]}
---

I also did this request with wget to see what the response should be, 
and it seems that this is the first part of the 297229 bytes long body. 
The response headers are:


---
  HTTP/1.1 200 OK
  Server: nginx
  Date: Fri, 19 Jul 2019 07:32:03 GMT
  Content-Type: application/json; charset=UTF-8
  Transfer-Encoding: chunked
  Vary: Accept-Encoding
  Vary: Accept-Encoding
  Cache-Control: private, must-revalidate
  ETag: "178c3f242b0151fe57e02f6e8817ce3a"
  Access-Control-Allow-Origin: *
  Access-Control-Allow-Methods: POST, GET, OPTIONS, PUT, PATCH, DELETE, 
HEAD

Length: unspecified [application/json]
---

Maybe the 'Length: unspecified' has something to do with it.

If I enable http-reuse the problem is still there. Only no option 
http-use-htx 'fixes' it.


I've stripped my config to the parts that I think are related:

---
global
master-worker
log /dev/loglocal0
log /dev/loglocal1 notice

daemon
userhaproxy
group   haproxy
maxconn 32768
spread-checks   3
nbproc  1
nbthread4
stats socket/var/run/haproxy.stat mode 666 level admin

	ssl-default-bind-ciphers 
ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS

ssl-default-bind-options no-sslv3 no-tls-tickets
	ssl-default-server-ciphers 
ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS

ssl-default-server-options no-sslv3 no-tls-tickets

tune.ssl.default-dh-param 2048

###
# Defaults
###
defaults
log global
timeout check   2s
timeout client  60s
timeout connect 10s
timeout http-keep-alive 4s
timeout http-request15s
timeout queue   30s
timeout server  60s
timeout tarpit  120s

errorfile 400 /etc/haproxy/errors.loc/400.http
errorfile 403 /etc/haproxy/errors.loc/403.http
errorfile 500 /etc/haproxy/errors.loc/500.http
errorfile 502 /etc/haproxy/errors.loc/502.http
errorfile 503 /etc/haproxy/errors.loc/503.http
errorfile 504 /etc/haproxy/errors.loc/504.http

frontend webservices
bind x.x.x.x:80 transparent
	bind x.x.x.x:443 transparent ssl crt /etc/haproxy/ssl/somecert.pem alpn 
h2,http/1.1

bind 2001:xxx:xxx:x::xx:80 transparent
	bind 2001:xxx:xxx:x::xx:443 transparent ssl crt 
/etc/haproxy/ssl/somecert.pem alpn h2,http/1.1


modehttp
maxconn 4096

option  httplog
option  dontlog-normal
option http-ignore-probes