I have a couple of questions regarding http-reuse and idle connections.
I'm referring to haproxy 2.6 ( docs.haproxy.org
https://docs.haproxy.org/2.6/configuration.html ) unless explicitly stated
otherwise. 1. Haproxy documentation ( docs.haproxy.org
https://docs.haproxy.org/2.6/configuration.html#http-reuse ) states that
specific modes differ in how they handle the first request of a session. What
is exactly meant by "a session" here and why is it so important if we
want to be on the safe side that _the first request_ is handled in a specific
way, not all the subsequent requests? 2. Second thing I want to ask is
related to the first, the documentation states in the description of safe mode
of http-reuse that "This ensures that in case the server closes the
connection when the request is being sent, the browser can decide to silently
retry it" Should it be understood that browsers have a mechanism which
allow them to retry requests if they fail (fail because of TCP connection
closure) but only on the condition that it is a first request of a session? Or
should it be understood that browsers have this mechanism and it applies to all
failed requests but we care about it only if the first request of a session can
be retried? If the second, why we don't care about subsequent requests?
Also why a browser wouldn't be able to retry a failed request if it has
occured when using http-reuse set to "aggressive" or "always"?
Will such request failures be logged in haproxy and if yes how can I recognize
them? Well, thinking about it more after I've written the above it seems
that in the direct client <-> server model (no proxy in the middle) it is
not possible for "the first request of a session" to be dispatched over
an existing TCP connection because such a connection simply has not been
established yet. But I'm not sure how it translates to client <->
haproxy <-> server case. 3. Once more about the phrase I have already
quoted: "This ensures that in case the server closes the connection when
the request is being sent (...)" As I understand it, risks related to
connection reuse is an inherent race condition that one side starts sending
request over a given TCP connection and another side already closes either this
(client - haproxy) or the related (haproxy - server) TCP connection. But here
I'm not sure about details. That is, I wonder what type of race conditions
exactly are possible and which of them are relevant to cases of http-reuse set
to the mode other than "never". And so we have the following cases:
i. the client sends a request over a given TCP connection to haproxy and
haproxy closes this TCP connection before that requests arrives, ii. the
client sends a request over a given TCP connection to haproxy and backend
closes the TCP connection that so far has been used to delivering requests to
it from haproxy. There are two subpossibilites here: a. that backend's
closure of the mentioned TCP connection happens earlier than the request
arrives to haproxy b. that backend's closure of the mentioned TCP
connection happens after that request's arrival to haproxy and here we have
still further possibilites: - haproxy learns about the closure of the
mentioned TCP connection before it started sending that request, - haproxy
learns about the closure of the mentioned TCP connection in the middle of
sending that request - haproxy learns about the closure of the mentioned TCP
connection after it has sent that request, I'm not sure if i) does have
negative consequences, does it? Will such a request be simply retried by a
browser? Well, I'm basically repeating what I asked in the point 2) but I
added it here for completeness. It seems that the case iia) has no negative
consequences because haproxy will simply either initiate a new TCP connection
to backend or it will use another existing one. Am I right here? Do negative
consequences occur in the case of iib)? Thinking about it, they should not
regadless of which a possibility given by dashes occurs - because if the first
dash occurs haproxy will simply recognize that the backend server has closed
the connection and it also will send this request over a new or an already
existing TCP connection. The second dash should not occur as it would be
illogical for the backend to close the connection when it is in a middle of
receiving a request (unless there is a long interruption in receiving packets
from haproxy and that silence hits a backend's timeout but it would suggest
a lossy network connection or similar things and such behaviours are anyway
expected then). In the last case it also seems illogical for backend to close
the TCP connection after it has already received a request without sending a
response. 4. What should be understood by "idle connections" in this
context if haproxy operates in http mode (is there a difference between what
they mean in http and tcp mode at all?)? Are these TCP connections over which
just nothing has been sent for a specified time or are these TCP connections
over which the conditions must be satisfied that - first - nothing has been
sent for a specified time and - second - there are no open HTTP/2 streams at
all (also no waiting for ACK packet from the second side at so on, in other
words the connection is truly silent both actually and potentially)? The
fragment of documentation that I quoted in the point above seems to suggest
that simply silence is sufficient and it is not significant if there are open
streams or not, but I'd like to be sure. 5. How long must silence elapse
in order for a TCP connection to be recognized to be "idle"? haproxy
makes tune.idletimer parameter available on which we read that (
docs.haproxy.org https://docs.haproxy.org/2.6/configuration.html#tune.idletimer
) it "Sets the duration after which HAProxy will consider that an empty
buffer is probably associated with an idle stream". Should it be understood
that it specifies how long silence on a TCP connection must elapse in order for
it to be recognized as "idle"? What is meant by "a stream"
here? 6. Parameter tune.pool-high-fd-ratio . Quoting from the documentation:
"Set a low threshold on the number of idling connections for a server,
below which a thread will not try to steal a connection from another
thread." The comma makes it a bit unclear for me if this parameter makes
haproxy maintain the number of idling connections so that it would be (after
this number has already been reached) never smaller than this value or it is
only a threshold for threads such that they can steal a connection from another
thread only when the number of idle connections is greater or equal to this
threshold? I've experimented with that and my experiments show that the
second interpretation holds true but I ask here to be sure. 7. Are there some
risks related to increasing pool-purge-delay to a high value - like, let's
say 25 minutes? From what I can think of it the risk is that at a moment there
will be so many connections that they will hit some haproxy or OS level limit
and creating new connections will not be possible. But it should only be a
problem if needed connections would require other parameters (in sense of
parameters that are needed in order for request to be dispatched over an
existing connection accorind to http-reuse documentation) than existing idle
ones, right? There could also be a risk of exhausting file descriptors but as I
can see it can be controlled by tune.pool-high-fd-ratio and
tune.pool-low-fd-ratio although I guess that when pool-purge-delay is low these
file descriptors can perhaps be used more effectively. By the way, in case in
which tune.pool-high-fd-ratio is exceeded, killing idle connections will be
realized (almost) immediately or only with purging happening as specified by
pool-purge-delay? 8. In the description of http-reuse safe mode it is stated
that: "There is also a special handling for the connections using
protocols subject to Head-of-line blocking (backend with h2 or fcgi). In this
case, when at least one stream is processed, the used connection is reserved to
handle streams of the same session. When no more streams are processed, the
connection is released and can be reused". I interpret what I quoted in
this case in this way: "if we have a TCP connection from haproxy to backend
which currently is processing at least one HTTP/2 stream, then the used
connection is reserved to handle streams of the same session". As I wrote
above I'm not quite sure what a session means in this context but from my
experiments it seems that it simply means the set of requests from the same IP
address and the reason for which we enforce that a given TCP connection has
only streams initiated by one client's IP is to avoid the situation in
which (I'm basically quoting here from docs.haproxy.org
https://docs.haproxy.org/2.8/configuration.html#http-reuse ) a mechanism known
as "head of line blocking" cause cascade effect on download speed for
all clients sharing a connection. I understand what is HoL blocking and that
imposes a problem in HTTP/2 when we have a lossy network because one lost
packet in a TCP connection causes all streams in this connection having to wait
for the successful retransmission of this packet (while for comparison, in case
of HTTP/1.1 with multiple parallel connections the requests / responses which
have being delivering over the other connections would simply continue being
transmitted as well as it'd possible to deliver new requests over them).
However, I wonder how exactly this mechanism affect client <-> haproxy
<-> server case. That is, as I imagine it we have a set of clients that
connect to haproxy and haproxy deliver their requests to backend over the same
TCP connection. There are many streams in this connection and each stream
consists of many frames. Now, sending frames from A to B is (in general)
realized in parallel to receiving frames from B to A but at any given moment
only a single frame is being transmitted over connection. So all further frames
are blocked by this frame. Now, if a backend server sends a frame to haproxy
and this haproxy has many clients and if we have a client which is slow to
download frame then why is it problem to other clients? That is, it seems to me
that haproxy in such a situation should transmit this frame to the client and
this particular download will be slow but at the same time it will also be
downloading the further frames from backend in that TCP connection and send
these other frames to other clients. Or it is not done this way because it
would violate www.rfc-editor.org https://www.rfc-editor.org/rfc/rfc9113.html
"Recipients process frames in the order they are received"? I also
asked chat GPT about this question and it answered that it is about TCP buffer
for this TCP connection between haproxy and backend filling up. Is the chat
right here?
