Hi Luke.

Am 21.01.2019 um 10:30 schrieb Luke Seelenbinder:
> Hi all,
> One more bug (or configuration hole) from our transition to 1.9.x using 
> end-to-end h2 connections.
> After enabling h2 backends (technically `server … alpn h2,http/1.1`), we 
> began seeing a high number of backend /server/ connection resets. A 
> reasonable number of client-side connection resets due to timeouts, etc., is 
> normal, but the server connection resets were new.
> I believe the root cause is that our backend servers are NGINX servers, which 
> by default have a 1000 request limit per h2 connection 
> (https://nginx.org/en/docs/http/ngx_http_v2_module.html#http2_max_requests). 
> As far as I can tell there's no way to set this to unlimited. That resulted 
> in NGINX resetting the HAProxy backend connections and thus resulted in user 
> requests being dropped or returning 404s (oddly enough; though this may be as 
> a result of the outstanding bug related to header manipulation and HTX mode).

Do you have such a info in the nginx log?

"http2 flood detected"

It's the message from this lines


> This wouldn't be a problem if one of the following were true:
> - HAProxy could limit the number of times it reused a connection

Can you try to set some timeout values for `timeout http-keep-alive`

I assume that this timeout could be helpful because of this block in the doc


  - KAL : keep alive ("option http-keep-alive") which is the default mode : all
    requests and responses are processed, and connections remain open but idle
    between responses and new requests.

and this code part


> - HAProxy could retry a failed request due to backend server connection reset 
> (possibly coming in 2.0 with L7 retries?)

Mind you to create a issue for that if there isn't one already?

> - NGINX could set that limit to unlimited.

Isn't `unsigned int` not enought ?
How many idle connections do you have for how long time?

> Our http-reuse is set to aggressive, but that doesn't make much difference, I 
> don't think, since safe would result in the same behavior (the connection is 
> reusable…but only for a limited number of requests).
> We've worked around this by only using h/1.1 on the backends, which isn't a 
> big problem for us, but I thought I would raise the issue, since I'm sure a 
> lot of folks are using haproxy <-> nginx pairings, and this is a bit of a 
> subtle result of that in full h2 mode.

Can you try to increase the max-requests to 200000 in nginx

The `max_requests` is defined as `ngx_uint_t` which is `unsigned int`

I have found this in the nginx source.


> Thanks again for such great software—I've found it pretty fantastic to run in 
> production. :)

Just for my curiosity, have you seen any changes for your solution with the htx
/H2 e2e?

> Best,
> Luke

Best regards

> —
> Luke Seelenbinder
> Stadia Maps | Founder
> stadiamaps.com

Reply via email to