Hi Willy,
I finally had the opportunity to try out `option abortonclose`.
Initially, it made the problem much worse. Instead of occasionally incorrect
status codes in the logs, I saw requests fail in the following manner:
[18/Mar/2019:12:30:08.040] stadiamaps~ tile/tile1 0/18603/-1/-1/24804 503 0 - -
sC-- 2/1/1/1/3 0/0 {} "GET /tiles/osm_bright/6/31/20.png HTTP/2.0"
[18/Mar/2019:12:30:08.041] stadiamaps~ tile/tile1 0/18602/-1/-1/24803 503 0 - -
sC-- 2/1/0/0/3 0/0 {} "GET /tiles/osm_bright/6/34/20.png HTTP/2.0"
What's further interesting, it is was consistently 2 out of 18 requests. That
led me down the road of checking queue timeouts (noticing the timing
correlation in the logs). I adjusted `timeout connect` up from 6200ms to
12400ms and added pool-purge-delay to 60s.
After adjusting those timeouts and pool purges and re-enabling `abortonclose`,
the request errors I was seeing magically went away. I'll push this config to
production and see if we see a reduction in 503s. I also suspect we'll see a
marginal improvement in throughput and response time due to keeping backend
connections open longer.
I'll also keep an eye our for inconsistencies between our backend accept
capability and timeouts and see if perhaps we're overrunning some buffer
somewhere in HAProxy, NGINX, or somewhere else.
Thanks for your help so far!
Best,
Luke
—
Luke Seelenbinder
Stadia Maps | Founder
stadiamaps.com
On Mon, Mar 4, 2019, at 14:08, Willy Tarreau wrote:
> On Mon, Mar 04, 2019 at 11:45:53AM +0000, Luke Seelenbinder wrote:
> > Hi Willy,
> >
> > > Do you have "option abortonclose" in your config ?
> >
> > We do not have abortonclose. Do you recommend this if we have a lot of
> > client-side request aborts (but not connection level closes)? From reading
> > the docs, I came away conflicted as to the implications. :-)
>
> It will help, especially if you have maxconn configured on your server
> lines, as it will allow the requests to be aborted while still in queue.
>
> That said, we still don't know exactly what causes your logs.
>
> Willy
>
