Re: Capture entire HTTP request (all headers one shot)

[Manas Gupta]

Hi Willy
Other captures work.

Here are the examples
https://gist.github.com/manasg/7f4d674a5e07b140e170eadad6858a24

Length 0 in req.payload did not change anything. Also isn't this dump a
binary blob?

I am open to trying the Lua approach.



On Fri, Sep 16, 2016 at 11:11 PM, Willy Tarreau  wrote:

> On Fri, Sep 16, 2016 at 06:08:40PM -0700, Manas Gupta wrote:
> > Thanks for the info Willy.
> >
> > Is this a layer 6 capture? If yes, do I need to do something special to
> log
> > it?
> >
> > The following snippet logs just a '-'
> >
> >   capture request header Host len 40
> >   capture request header Via len 100
> >   capture request header xyz len 52
> >   http-request capture req.payload(0,0) len 4000
> >
> > In log format :-
> >
> > hdump=%[capture.req.hdr(3)]
>
> Does it work if you capture something else ? And if you force a length
> instead of 0 in req.payload ? I'm just trying to figure what could cause
> the problem.
>
> Willy
>

srv_conn vs be_conn

[Dmitry Sivachenko]

Hello,

I have few questions:

1) in documentation about srv_conn we have:
---
Returns an integer value corresponding to the number of currently established 
connections on the designated server, possibly including the connection being 
evaluated. 


What does it mean "including the connection being evaluated"?

2) is it true that be_conn(B) == sum(srv_conn(B/srv)) for each srv in backend B?

3) Does srv_conn(srv) equals to what I see in Sessions->Current in haproxy 
stats page for that server?

Thanks in advance.

[PATCH 2/4] Compute a "cumulative weight" to allow chash balancing to hit its target

[Andrew Rodland]

For active servers, this is the sum of the eweights of all active
servers before this one in the backend, and
[srv->cumulative_weight .. srv_cumulative_weight + srv_eweight) is a
space occupied by this server in the range [0 .. lbprm.tot_wact), and
likewise for backup servers with tot_wbck. This allows choosing a
server or a range of servers proportional to their weight, by simple
integer comparison.
---
 include/types/server.h | 1 +
 src/backend.c  | 2 ++
 2 files changed, 3 insertions(+)

diff --git a/include/types/server.h b/include/types/server.h
index ce13820..5d89212 100644
--- a/include/types/server.h
+++ b/include/types/server.h
@@ -203,6 +203,7 @@ struct server {
unsigned wscore;/* weight score, used during 
srv map computation 
*/
unsigned prev_eweight;  /* eweight before last change */
unsigned rweight;   /* remainer of weight in the 
current LB tree 
*/
+   unsigned cumulative_weight; /* weight of servers prior to 
this one in 
the same group, for chash balancing */
unsigned npos, lpos;/* next and last positions in 
the LB tree */
struct eb32_node lb_node;   /* node used for tree-based 
load 
balancing */
struct eb_root *lb_tree;/* we want to know in what tree 
the server is */
diff --git a/src/backend.c b/src/backend.c
index faf872c..573f054 100644
--- a/src/backend.c
+++ b/src/backend.c
@@ -115,9 +115,11 @@ void recount_servers(struct proxy *px)
!(px->options & PR_O_USE_ALL_BK))
px->lbprm.fbck = srv;
px->srv_bck++;
+   srv->cumulative_weight = px->lbprm.tot_wbck;
px->lbprm.tot_wbck += srv->eweight;
} else {
px->srv_act++;
+   srv->cumulative_weight = px->lbprm.tot_wact;
px->lbprm.tot_wact += srv->eweight;
}
}
-- 
2.9.3

[PATCH 3/4] Add hash-balance-factor option for hash-type consistent

[Andrew Rodland]

0 will mean no balancing occurs; otherwise it represents the ratio
between the highest-loaded server and the average load, times 100 (i.e.
a value of 150 means a 1.5x ratio), assuming equal weights.
---
 include/types/lb_chash.h |  1 +
 src/cfgparse.c   | 13 +
 2 files changed, 14 insertions(+)

diff --git a/include/types/lb_chash.h b/include/types/lb_chash.h
index 5991ce9..b711636 100644
--- a/include/types/lb_chash.h
+++ b/include/types/lb_chash.h
@@ -30,6 +30,7 @@ struct lb_chash {
struct eb_root act; /* weighted chash entries of active servers */
struct eb_root bck; /* weighted chash entries of backup servers */
struct eb32_node *last; /* last node found in case of round robin (or 
NULL) */
+   int balance_factor; /* load balancing factor * 100, 0 if disabled */
 };
 
 #endif /* _TYPES_LB_CHASH_H */
diff --git a/src/cfgparse.c b/src/cfgparse.c
index 7b58ef6..376c8b0 100644
--- a/src/cfgparse.c
+++ b/src/cfgparse.c
@@ -5932,6 +5932,19 @@ stats_error_parsing:
}
}
}
+   else if (strcmp(args[0], "hash-balance-factor") == 0) {
+   if (*(args[1]) == 0) {
+   Alert("parsing [%s:%d] : '%s' expects an integer 
argument.\n", file, linenum, args[0]);
+   err_code |= ERR_ALERT | ERR_FATAL;
+   goto out;
+   }
+   curproxy->lbprm.chash.balance_factor = atol(args[1]);
+   if (curproxy->lbprm.chash.balance_factor != 0 && 
curproxy->lbprm.chash.balance_factor <= 100) {
+   Alert("parsing [%s:%d] : '%s' must be 0 or greater than 
100.\n", file, linenum, args[0]);
+   err_code |= ERR_ALERT | ERR_FATAL;
+   goto out;
+   }
+   }
else if (strcmp(args[0], "unique-id-format") == 0) {
if (!*(args[1])) {
Alert("parsing [%s:%d] : %s expects an argument.\n", 
file, linenum, args[0]);
-- 
2.9.3

[PATCH 4/4] Implement bounded-load hash algorithm

[Andrew Rodland]

The consistent hash lookup is done as normal, then if balancing is
enabled, we progress through the hash ring until we find a server that
doesn't have "too much" load. In the case of equal weights for all
servers, the allowed number of requests for a server is either the
floor or the ceil of (num_requests * hash-balance-factor / num_servers);
with unequal weights things are somewhat more complicated, but the
spirit is the same -- a server should not be able to go too far above
(its relative weight times) the average load. Using the hash ring to
make the second/third/etc. choice maintains as much locality as
possible given the load limit.
---
 src/lb_chash.c | 39 ++-
 1 file changed, 38 insertions(+), 1 deletion(-)

diff --git a/src/lb_chash.c b/src/lb_chash.c
index a62dfb5..401e36e 100644
--- a/src/lb_chash.c
+++ b/src/lb_chash.c
@@ -242,6 +242,31 @@ static void chash_update_server_weight(struct server *srv)
 }
 
 /*
+ * This function implements the "Consistent Hashing with Bounded Loads" 
algorithm
+ * of Mirrokni, Thorup, and Zadimoghaddam (arxiv:1608.01350), adapted for use 
with
+ * unequal server weights.
+ */
+int chash_server_is_eligible(struct server *s) {
+  /* The total number of slots to allocate is the total number of outstanding 
requests 
+   * (including the one we're about to make) times the load-balance-factor, 
rounded up.
+   */
+   unsigned tot_slots = ((s->proxy->served + 1) * 
s->proxy->lbprm.chash.balance_factor + 99) / 100;
+   unsigned slots_per_weight = tot_slots / s->proxy->lbprm.tot_weight;
+   unsigned remainder = tot_slots % s->proxy->lbprm.tot_weight;
+
+  /* Allocate a whole number of slots per weight unit... */
+   unsigned slots = s->eweight * slots_per_weight;
+  /* And then distribute the rest among servers proportionally to their 
weight. */
+   slots += ((s->cumulative_weight + s->eweight) * remainder) / 
s->proxy->lbprm.tot_weight
+   - (s->cumulative_weight * remainder) / 
s->proxy->lbprm.tot_weight;
+  /* But never leave a server with 0. */ 
+   if (slots == 0)
+   slots = 1;
+
+   return s->served < slots;
+}
+
+/*
  * This function returns the running server from the CHASH tree, which is at
  * the closest distance from the value of . Doing so ensures that even
  * with a well imbalanced hash, if some servers are close to each other, they
@@ -287,7 +312,19 @@ struct server *chash_get_server_hash(struct proxy *p, 
unsigned int hash)
dp = hash - prev->key;
dn = next->key - hash;
 
-   return (dp <= dn) ? psrv : nsrv;
+   if (dp <= dn) {
+   next = prev;
+   nsrv = psrv;
+   }
+
+   while (p->lbprm.chash.balance_factor && 
!chash_server_is_eligible(nsrv)) {
+   next = eb32_next(next);
+   if (!next)
+   next = eb32_first(root);
+   nsrv = eb32_entry(next, struct tree_occ, node)->server;
+   }
+
+   return nsrv;
 }
 
 /* Return next server from the CHASH tree in backend . If the tree is empty,
-- 
2.9.3

transparent or intercepting proxy with https

[Brendan Kearney]

i am trying to setup a transparent or intercepting proxy, that works 
with HTTPS, and have hit a bit of a wall.


i am using IPTables to intercept the port 80 and 443 traffic, and 
DNAT'ing the traffic to a HAProxy VIP.


i have the front end configured as such:

frontend tproxy
bind 192.168.120.1:3129
option httplog
option http-server-close
option forwardfor except 127.0.0.0/8
default_backend tproxy

the backend is where i have problems.

backend tproxy
acl https ssl_fc

http-request set-uri http://%[req.hdr(Host)]%[path]?%[query] 
unless https


http-request set-method CONNECT if https
http-request set-uri https://%[ssl_fc_sni] if https

server proxy1 192.168.88.1:3129 check inter 1
server proxy2 192.168.88.2:3129 check inter 1

right now, HTTP interception works without issue.  as i understand 
things having read through some docs, the acl will never match HTTPS 
traffic that is to be proxied, because the front end bind statement does 
not have the "ssl" option.  subsequently, the rewrites of the method and 
uri will never happen.  i also believe the rewrite of the uri will not 
work because ssl_fc_sni requires the "ssl" option be present on the bind 
line for the front end.  that leads me to wonder how i differentiate 
between HTTP and HTTPS in a transparent proxy scenario.  would 
req.proto_http be appropriate?  being that the match does not occur 
until the request is complete, i am not sure.


once i am properly differentiating between HTTP and HTTPS traffic, what 
would the correct way to rewrite the uri?  i think req.ssl_sni is the 
value i need to use, instead of ssl_fc_sni.


any insight is appreciated.

thank you,

brendan

Re: resolvers section: nameservers required?

[Baptiste]

On Thu, Sep 15, 2016 at 10:53 PM, Topher Cullen 
wrote:

> The documentation is unclear to me which, if any, of the resolver options
> are required for run time resolution. I defined a resolver section like
> this:
>
> resolver mydns
>   hold valid 2s
>
> I then configured a backend to use it. haproxy started without error or
> warning, but I don't believe its actually doing resolution. Can anyone shed
> so light on this new feature?
>
>
Hi,

Technically speaking, if there is no nameserver in the resolvers section,
then HAProxy won't perform any DNS resolution at run time.

I think you're right and we should report a Warning in such case.

Baptiste

Re: [PATCH] MAJOR: filters: Add filters support

[Willy Tarreau]

Hi Bertrand,

On Tue, Sep 20, 2016 at 12:13:32AM +0100, Bertrand Jacquin wrote:
> > And finally, If you can share with me your HA and
> > Nginx configurations, this could help.
> 
> I'm attaching a strip down version of haproxy/nginx/php-fpm on which I
> can reproduice this issue.

I think another thing would be extremely useful, it would be a full-packet
network capture between haproxy and nginx so that we have the full headers,
the chunk sizes (if any) and the response timing, which generally matters a
lot. Ideally the dump in text (or binary) format in a distinct file using
curl -i would be nice as well.

Thanks!
Willy

Re: haproxy - namespece implementation and usage

[Willy Tarreau]

Hi Martin,

On Sat, Sep 17, 2016 at 11:16:15PM +0200, Martin Tóth wrote:
> Hi fellow haproxy users,
> 
> i just wanted to ask if new implementation of haproxy (implemented in v.
> 1.6.9) namespaces can work like this. I have Zabbix proxy daemon running
> inside network namespace in Linux, let???s say namespace is named
> ???customer???.
> I want to be able to run haproxy daemon in default linux namespace and be
> able to connect with haproxy to Zabbix proxy demon running inside own
> namespace. Is this possible ?
> 
> My config :
> 
> namespace_list
>   namespace customer
> 
> frontend customer
>   mode tcp
>   bind 10.0.0.2:10001 accept-proxy # this is IP and port on host 
> (10.0.0.2 - linux server IP) where i should connect when i want to reach 
> customer Zabbix proxy daemon
>   default_backend serverlist
> 
> backend serverlist
>   mode tcp
>   server s1 10.8.1.4:10050 namespace customer # this is zabbix proxy 
> dameon
> 
> It did not found any related example of configuration or more than one page
> of documentation. 

It should work like this. I've used it just for a test recently, to see
if it was possible to isolate a daemon into a network-less namespace (no
NIC except "lo"), and with haproxy connecting into that namespace. And
yes, it works. I'm not a namespace user at all so I had to use "ip netns"
with the man page opened and after trial and error I managed to make it
work.

At least you need to enter your customer namespace and issue
"netstat -ltnp" to ensure that your zabbix server is properly listening
to incoming connections, otherwise it will obviously never work.

Hoping this helps,
Willy

Re: envoy LB is now an open source project

[Willy Tarreau]

Hi, Pavlos,

On Thu, Sep 15, 2016 at 01:09:16AM +0200, Pavlos Parissis wrote:
> Hi,
> 
> It is a very interesting project, https://lyft.github.io/envoy/
> 
> Here is a comparison with HAProxy
> https://lyft.github.io/envoy/docs/intro/comparison.html

Thanks for the link, it sounds interesting. It looks very young however,
and given that oldest bugs in various load balancers can be around 10
years old with most important ones up to 5 years old, it may take a bit
of time to cool down but regardless of this it looks very interesting.

Cheers,
Willy

Re: [PATCH] MINOR: enable IP_BIND_ADDRESS_NO_PORT on backend connections

[Willy Tarreau]

Hi Pavlos,

On Wed, Sep 14, 2016 at 11:01:36PM +0200, Pavlos Parissis wrote:
> in our setup where we have haproxy in PoPs which forwards traffic to haproxy
> servers in main data-centers, I am planning to address the ephemeral port
> exhaustion symptom by having the frontends in data centers listening on 
> multiple
> IPs, so I can have the same server multiple times in the backend at PoP.
> 
> backend data_center_haproxies
>   server1_on_ip1 1.1.1.1 
>   server1_on_ip2 1.1.1.2 
> 
> with our system inventory/puppet infra assigning multiple IPs on servers at 
> PoP
> isn't that simple, I know it sounds weird.

Note that you can also make your servers listen on multiple ports, or use
multiple addresses on haproxy for this. I tend to prefer having multiple
ports because it multiplies the allocatable port ranges without adding IP
addresses anywhere.

Another point to note is that if you're running out of source ports due
to idle keep-alive connections between haproxy and the servers, you can
enable http-reuse to significantly improve the situation. It will also
remove one round-trip for the connect() and will reduce the memory usage
on the server side, so there are benefits everywhere.

Regards,
Willy