Re: fullconn not working

[Patrick Hemmer]

*From:* Pavlos Parissis [mailto:pavlos.paris...@gmail.com]
*Sent:* Tuesday, July 16, 2019, 09:32 EDT
*To:* haproxy@formilux.org
*Cc:* Patrick Hemmer 
*Subject:* fullconn not working


On Παρασκευή, 28 Ιουνίου 2019 5:50:48 Μ.Μ. CEST Patrick Hemmer wrote:

I'm trying to get fullconn working, and can't seem to do so. I dunno if
it's a bug, or if it's my understanding that's wrong.
Basically my goal is to prevent the cumulative total of all connections
to all servers in a pool from exceeding a certain value.
For example I might have 10 servers, each with a maxconn of 10. But I
want to configure haproxy with a pool-wide limit of 50, so that even if
the connections are well distributed and no one server is maxed out,
after 50 connections to all servers, haproxy will still start to queue
instead.

fullconn seems like the right way to accomplish this, however I cannot
get it to work. I've tried a simple setup of 2 servers, each with
`maxconn 3`, and then a backend `fullconn 2`, which should result in
queuing after 2 simultaneous connections, however it doesn't. If I send
4 connections, all 4 are simultaneously sent to the backend servers.

Here's my test config:
defaults
  log 127.0.0.1:1234 daemon
  mode http
  option httplog
  timeout queue 5s
frontend f1
  bind :8001
  default_backend b1
backend b1
  fullconn 2
  server s1 127.0.0.1:8081 minconn 1 maxconn 3
  server s2 127.0.0.1:8081 minconn 1 maxconn 3

Here's how I test:
for i in {1..4}; do curl "http://localhost:8001/?sleep=2=$i; & done

And here's the logs:
<30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55119
[28/Jun/2019:11:37:45.658] f1 b1/s2 0/0/0/2003/2003 200 75 - - 
4/4/3/2/0 0/0 "GET /?sleep=2=3 HTTP/1.1"
<30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55117
[28/Jun/2019:11:37:45.658] f1 b1/s2 0/0/0/2003/2003 200 75 - - 
4/4/2/1/0 0/0 "GET /?sleep=2=4 HTTP/1.1"
<30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55118
[28/Jun/2019:11:37:45.658] f1 b1/s1 0/0/0/2003/2003 200 75 - - 
4/4/1/2/0 0/0 "GET /?sleep=2=1 HTTP/1.1"
<30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55120
[28/Jun/2019:11:37:45.658] f1 b1/s1 0/0/0/2003/2003 200 75 - - 
4/4/0/1/0 0/0 "GET /?sleep=2=2 HTTP/1.1"



Your e-mail client mangled above log lines and as a result they are bit 
unreadable.
The 4th field from `4/4/3/2/0`  is srv_conn  *2* which is below the maxconn of 
*3*, so haproxy did
the right thing as it didn't allow more than *full_conn* connections to be 
concurrently opened
against the server.

Cheers,
Pavlos

maxconn and fullconn are different settings.
maxconn: 
https://cbonte.github.io/haproxy-dconv/1.9/configuration.html#maxconn%20(Server%20and%20default-server%20options)
fullconn: 
https://cbonte.github.io/haproxy-dconv/1.9/configuration.html#fullconn


-Patrick

[RFC] MINOR: attempt to insert into priority queue when full

[Patrick Hemmer]

This is a draft patch which essentially allows a high priority request 
to replace a low priority request when the queue is full.


Currently if a high priority request comes along, and the destination 
queue is full, the request gets rejected (or redispatched if possible). 
This change allows inserting into the queue if there are lower priority 
requests present. When doing so, the lowest priority request is evicted 
to make room.


The intent is for that evicted request to be handled as if it just came 
in. Meaning if there were a brand new request coming in, and the 
configuration were such that the request should be rejected, then the 
evicted request should be rejected. If redispatch is enabled a new 
request would be sent to a different server, and so the evicted request 
should be as well.



Now as for why this is tagged RFC: The implementation for the handling 
of the evicted request is bad. If you notice inside pendconn_displace(), 
the section of code that handles this (lines 498-505) has no business 
being there. Those operations should be handled higher up. It also 
doesn't work as intended, as instead of responding to the client with a 
503, it just kills the connection.


However I'm having difficulty finding the appropriate way to do this, 
and could use some guidance. I've tried several different things, and 
none of the behaviors are quite right.
The simplest solution of just performing __pendconn_unlink() and then 
waking the task results in the request being redispatched. The higher 
level code assumes that if the request was in a queue, and is now no 
longer in a queue, then redispatch is the appropriate action.


Thanks

-Patrick
From a3c8ba92a05ec877662359f963ece0cfa82051f8 Mon Sep 17 00:00:00 2001
From: Patrick Hemmer 
Date: Thu, 12 Sep 2019 22:56:51 -0400
Subject: [PATCH] MINOR: attempt to insert into priority queue when full

This makes it so that when a queue (server or proxy) is full, that we try to
insert into the queue and evict a request of lower priority.

The evicted request will either be redispatched if `option redispatch` is
enabled, or rejected if not.
---
 include/proto/queue.h |   1 +
 src/backend.c |   3 +-
 src/queue.c   | 119 ++
 3 files changed, 122 insertions(+), 1 deletion(-)

diff --git a/include/proto/queue.h b/include/proto/queue.h
index a7ab63b35..156bf3964 100644
--- a/include/proto/queue.h
+++ b/include/proto/queue.h
@@ -37,6 +37,7 @@
 extern struct pool_head *pool_head_pendconn;
 
 struct pendconn *pendconn_add(struct stream *strm);
+int pendconn_displace(struct stream *strm);
 int pendconn_dequeue(struct stream *strm);
 void process_srv_queue(struct server *s);
 unsigned int srv_dynamic_maxconn(const struct server *s);
diff --git a/src/backend.c b/src/backend.c
index 1b01536c1..56340d2de 100644
--- a/src/backend.c
+++ b/src/backend.c
@@ -976,7 +976,8 @@ int assign_server_and_queue(struct stream *s)
(srv->nbpend || srv->served >= srv_dynamic_maxconn(srv))) {
 
if (srv->maxqueue > 0 && srv->nbpend >= srv->maxqueue)
-   return SRV_STATUS_FULL;
+   // queue is full. see if priority allows us to 
insert
+   return pendconn_displace(s);
 
p = pendconn_add(s);
if (p)
diff --git a/src/queue.c b/src/queue.c
index 30b7ef056..ea015272a 100644
--- a/src/queue.c
+++ b/src/queue.c
@@ -185,6 +185,33 @@ void pendconn_unlink(struct pendconn *p)
pendconn_queue_unlock(p);
 }
 
+/* Comapres 2 pendconn queue priority keys.
+ *
+ * Returns :
+ *   -1 if k1 < k2
+ *   0  if k1 == k2
+ *   1  if k1 > k2
+ */
+static int key_cmp(u32 k1, u32 k2)
+{
+   if (KEY_CLASS(k1) < KEY_CLASS(k2))
+   return -1;
+   if (KEY_CLASS(k1) > KEY_CLASS(k2))
+   return 1;
+
+   if (k1 < NOW_OFFSET_BOUNDARY())
+   k1 += 0x10; // key in the future
+   if (k2 < NOW_OFFSET_BOUNDARY())
+   k2 += 0x10; // key in the future
+
+   if (k1 < k2)
+   return -1;
+   if (k1 > k2)
+   return 1;
+
+   return 0;
+}
+
 /* Retrieve the first pendconn from tree . Classes are always
  * considered first, then the time offset. The time does wrap, so the
  * lookup is performed twice, one to retrieve the first class and a second
@@ -212,6 +239,31 @@ static struct pendconn *pendconn_first(struct eb_root 
*pendconns)
return eb32_entry(node2, struct pendconn, node);
 }
 
+/* Retrieve the last pendconn from tree .
+ * Follows the same semantics as pendconn_first.
+ */
+static struct pendconn *pendconn_last(struct eb_root *pendconns)
+{
+   struct eb32_node *node, *node2 = NULL;
+   u32 key;
+
+   node = eb32_last(pendconns);
+   if (!node)
+   return NULL;
+
+   key = KEY_CL

Re: fullconn not working

[Patrick Hemmer]

*From:* Jerome Magnin [mailto:jmag...@haproxy.com]
*Sent:* Tuesday, July 16, 2019, 10:19 EDT
*To:* Patrick Hemmer 
*Cc:* Pavlos Parissis , haproxy@formilux.org
*Subject:* fullconn not working


Hi Patrick,

On Tue, Jul 16, 2019 at 09:40:31AM -0400, Patrick Hemmer wrote:


*From:* Pavlos Parissis [mailto:pavlos.paris...@gmail.com]
*Sent:* Tuesday, July 16, 2019, 09:32 EDT
*To:* haproxy@formilux.org
*Cc:* Patrick Hemmer 
*Subject:* fullconn not working


On Παρασκευή, 28 Ιουνίου 2019 5:50:48 Μ.Μ. CEST Patrick Hemmer wrote:

I'm trying to get fullconn working, and can't seem to do so. I dunno if
it's a bug, or if it's my understanding that's wrong.
Basically my goal is to prevent the cumulative total of all connections
to all servers in a pool from exceeding a certain value.
For example I might have 10 servers, each with a maxconn of 10. But I
want to configure haproxy with a pool-wide limit of 50, so that even if
the connections are well distributed and no one server is maxed out,
after 50 connections to all servers, haproxy will still start to queue
instead.

fullconn seems like the right way to accomplish this, however I cannot
get it to work. I've tried a simple setup of 2 servers, each with
`maxconn 3`, and then a backend `fullconn 2`, which should result in
queuing after 2 simultaneous connections, however it doesn't. If I send
4 connections, all 4 are simultaneously sent to the backend servers.

Here's my test config:
defaults
   log 127.0.0.1:1234 daemon
   mode http
   option httplog
   timeout queue 5s
frontend f1
   bind :8001
   default_backend b1
backend b1
   fullconn 2
   server s1 127.0.0.1:8081 minconn 1 maxconn 3
   server s2 127.0.0.1:8081 minconn 1 maxconn 3

Here's how I test:
for i in {1..4}; do curl "http://localhost:8001/?sleep=2=$i; & done

And here's the logs:
<30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55119
[28/Jun/2019:11:37:45.658] f1 b1/s2 0/0/0/2003/2003 200 75 - - 
4/4/3/2/0 0/0 "GET /?sleep=2=3 HTTP/1.1"
<30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55117
[28/Jun/2019:11:37:45.658] f1 b1/s2 0/0/0/2003/2003 200 75 - - 
4/4/2/1/0 0/0 "GET /?sleep=2=4 HTTP/1.1"
<30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55118
[28/Jun/2019:11:37:45.658] f1 b1/s1 0/0/0/2003/2003 200 75 - - 
4/4/1/2/0 0/0 "GET /?sleep=2=1 HTTP/1.1"
<30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55120
[28/Jun/2019:11:37:45.658] f1 b1/s1 0/0/0/2003/2003 200 75 - - 
4/4/0/1/0 0/0 "GET /?sleep=2=2 HTTP/1.1"



Your e-mail client mangled above log lines and as a result they are bit 
unreadable.
The 4th field from `4/4/3/2/0`  is srv_conn  *2* which is below the maxconn of 
*3*, so haproxy did
the right thing as it didn't allow more than *full_conn* connections to be 
concurrently opened
against the server.

Cheers,
Pavlos

maxconn and fullconn are different settings.
maxconn: 
https://cbonte.github.io/haproxy-dconv/1.9/configuration.html#maxconn%20(Server%20and%20default-server%20options)
fullconn:
https://cbonte.github.io/haproxy-dconv/1.9/configuration.html#fullconn

-Patrick

fullconn is not used to set a limit on the amount of connections handled by the
backend. It is used to have a 'dynamic' maxconn value on server lines.

this dymanic maxconn value will never be below minconn, and never higher than
maxconn. When the amount of connections handled by backend is between 0 and
fullconn, this dynamic maxconn value is somewhere between minconn and maxconn,
proportionnate to where we are between 0 and fullconn. When fullconn is reached,
dynamic maxconn equals the maxconn you set on server line.

With the values you set, when you reach 2 connections at backend level, servers
will allow 3 at most, each.

Jérôme

Thanks, I think I'm following now. My understanding was backwards.
I'm guessing the use case for this is so that you have maxconn at a 
value where servers have a good response time. But then if you start 
queueing too much, then maxconn is raised, trading slower response time 
for higher throughput.



So my next question, is there any way to set a backend connection limit 
(after which connections get queued)? I want to limit the number of 
active connections across the whole pool, regardless of the number of 
servers, or what the limit on each server is.


The reason is that there are shared resources (e.g. a database, NAS 
filesystem, etc) behind a pool, and that shared resource can only handle 
so much load. We can divide the cumulative limit across the number of 
servers in the pool, but if some of the servers are out of the pool, 
then that limit shrinks and we're not utilizing the full capacity of the 
shared resources.



-Patrick

Re: How to wait some time before retry?

[Patrick Hemmer]

*From:* Marco Colli [mailto:collimarc...@gmail.com]
*Sent:* Friday, September 27, 2019, 07:21 EDT
*To:* HAProxy 
*Subject:* How to wait some time before retry?

Still have this issue and I cannot find a solution. It would be great 
to have an option "wait time before retry" in the next versions of 
HAProxy (instead of the fixed value of 1 sec).


On Mon, Sep 16, 2019 at 2:03 PM Marco Colli wrote:


Hello!

I have a question about HAProxy configuration. Maybe someone has a
solution ;)

I have a HAProxy (v2.0) load balancer in front of many web servers.

When I restart the web servers the TCP socket remains closed for a
few seconds (~10s). For this reason I would like to retry failed
attempts to connect after some seconds.

I already use |option redispatch|, however it seems that does not
solve my issue. The problem is that the request is retried
immediately (after 1s), thus causing all the retries to fail. From
the HAProxy docs:

In order to avoid immediate reconnections to a server which is
restarting, a turn-around timer of min("timeout connect", one
second) is applied before a retry occurs.

Is there any option to wait some more time (e.g. 10s) before
retrying? Or do you have any other solution?




Not the cleanest solution, but something that could work would be to add 
a lua action to sleep for 10 seconds on the response when you have a 
connect error, and then override the response to a 307 (or perhaps 302) 
redirect back to the same location. This will then cause the browser to 
retry the request.



-Patrick

fullconn not working

[Patrick Hemmer]

I'm trying to get fullconn working, and can't seem to do so. I dunno if 
it's a bug, or if it's my understanding that's wrong.
Basically my goal is to prevent the cumulative total of all connections 
to all servers in a pool from exceeding a certain value.
For example I might have 10 servers, each with a maxconn of 10. But I 
want to configure haproxy with a pool-wide limit of 50, so that even if 
the connections are well distributed and no one server is maxed out, 
after 50 connections to all servers, haproxy will still start to queue 
instead.


fullconn seems like the right way to accomplish this, however I cannot 
get it to work. I've tried a simple setup of 2 servers, each with 
`maxconn 3`, and then a backend `fullconn 2`, which should result in 
queuing after 2 simultaneous connections, however it doesn't. If I send 
4 connections, all 4 are simultaneously sent to the backend servers.


Here's my test config:
defaults
    log 127.0.0.1:1234 daemon
    mode http
    option httplog
    timeout queue 5s
frontend f1
    bind :8001
    default_backend b1
backend b1
    fullconn 2
    server s1 127.0.0.1:8081 minconn 1 maxconn 3
    server s2 127.0.0.1:8081 minconn 1 maxconn 3

Here's how I test:
for i in {1..4}; do curl "http://localhost:8001/?sleep=2=$i; & done

And here's the logs:
<30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55119 
[28/Jun/2019:11:37:45.658] f1 b1/s2 0/0/0/2003/2003 200 75 - -  
4/4/3/2/0 0/0 "GET /?sleep=2=3 HTTP/1.1"
<30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55117 
[28/Jun/2019:11:37:45.658] f1 b1/s2 0/0/0/2003/2003 200 75 - -  
4/4/2/1/0 0/0 "GET /?sleep=2=4 HTTP/1.1"
<30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55118 
[28/Jun/2019:11:37:45.658] f1 b1/s1 0/0/0/2003/2003 200 75 - -  
4/4/1/2/0 0/0 "GET /?sleep=2=1 HTTP/1.1"
<30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55120 
[28/Jun/2019:11:37:45.658] f1 b1/s1 0/0/0/2003/2003 200 75 - -  
4/4/0/1/0 0/0 "GET /?sleep=2=2 HTTP/1.1"



So am I misunderstanding how fullconn works? Or is there a bug?
I've tested with 2.0.1, 1.9.8, and 1.8.13.

-Patrick

Re: [PATCH] improving github experience, kindly ask people to reproduce bugs on latest haproxy

[Patrick Hemmer]

*From:* Илья Шипицин [mailto:chipits...@gmail.com]
*Sent:* Thursday, September 19, 2019, 15:10 EDT
*To:* HAProxy 
*Subject:* [PATCH] improving github experience, kindly ask people to 
reproduce bugs on latest haproxy



hello,

please find attached patch

Ilya Shipitsin
I dunno, I've personally never been fond of it when bug reporters are 
blindly asked to upgrade to the latest version. Sometimes the request is 
justified, such as when the project maintainers have reason to believe 
the bug is fixed, or if the version is years old. But otherwise it can 
causes difficulties for the reporter. In corporate environments, it can 
be difficult to perform such upgrades. Sometimes these issues are only 
reproducible in production environments. So by asking them to upgrade, 
you're making them go through the difficulty, and potentially cause 
impact to their clients. And because that process can take a while, it's 
possible that by the time they do complete the upgrade, another version 
has been released.


I personally also find the use of heavy bug templates, with nuanced 
little checkboxes to be annoying. In this case more so because we 
already ask them to provide the version information, which answers the 
question the checkbox is for. And the checkbox "yes i'm on the latest" 
might be accurate at the time of submission, but can become out of date.


Now all that said, I'm not against encouraging people to try a later 
version if available. Just that I don't think it should be the expectation.


-Patrick