Hi Veiko,
On Thu, Feb 14, 2019 at 02:31:42PM +0000, Veiko Kukk wrote:
>
> On 2019-02-01 13:30, Veiko Kukk wrote:
> > On 2019-02-01 12:34, Aleksandar Lazic wrote:
> >
> > > Do you have any errors in lighthttpds log?
> >
> > Yes, it has error messages about not being enable to write to socket.
> >
> > Unrecoverable error writing to socket! errno 32, retries 120000, ppoll
> > return 1, send return -1
> > ERROR: Couldn't write header data to socket! desired: 4565 / actual: -1
> >
> > I've tested with several hundred thoused requests, but it never
> > happens when using "mode tcp".
>
> Replaced HAproxy with Nginx for testing and with Nginx, not a single
> connection was interrupted, did millions of requests.
This is interesting. As you observed in the trace you sent me, the
lighttpd server closes just after sending the response headers. This
indeed matches the "SD" log that aproxy emits. If it doesn't happen
in TCP mode nor with Nginx, it means that something haproxy modifies
in the request causes this effect on the server.
The request you isolated looks like this :
GET /video/XXXX_redacted_XXXX HTTP/1.1
Host: 127.0.0.1
X-Request-Id: XXX_redacted_XXX
Connection: close
X-Forwarded-For: 127.0.0.1
So here what haproxy has apparently modified is :
- the X-Request-Id header field that was appended
- the "Connection: close" header field because you likely have
"option http-server-close" in the config ;
- the "X-Forwarded-For:" header field (option forwardfor)
Among them I hardly see why the request-id or XFF could cause any issue,
but we could imagine that for any reason the server occasionally has
trouble with "connection: close".
If you have "option http-server-close" in your config, it would be nice
to remove it and see if it changes anything. If you don't have it because
the request already arrives with close (but then I hardly understand how
it would work fine in TCP mode), then you could try to add
"option http-pretend-keepalive" so that haproxy removes this header
before passing the request to the server.
If you could also share a capture of one request forwarded by Nginx to
the server it would be nice. We may find a minor difference which could
tell us what makes lighttpd behave badly here.
Cheers,
Willy
