Improve config with lots of distinct matching types

[Joao Morais]

Hello list, tl;dr is: How can I avoid configuring dozens of http-request with 
the same acl?

My use case is a haproxy cluster receiving requests for hundreds of distinct 
hostnames, several of them with a dozen or so distinct paths, and a few more 
than 5k distinct backends that sends these requests to a cluster of about 15k 
containers. In the very beginning we were adding one http-request per rule but 
this wasn't scaling well, so changed the approach to a map() converter with 
something like `# ` which improved performance a 
lot.

Now we're starting to add other dimensions to that host+path matching, which is 
at least query params and header matching, so starting again with one 
http-request keyword per rule, which we already know that will not scale.

My starting idea was to evolve the map() converter strategy to add these new 
matching rules but couldn't figure out a way to do it. I thought about making 
two levels of map() but it doesn't allow dynamic input, so I wouldn't have a 
way to configure the second one.

Another idea was to mimic how other proxies do their configuration: an outer 
block defines the hostname level, everything inside that block already have an 
implicit `{ hdr(host) host1 }`. A second block within the first one defines the 
path, so everything inside it has already an implicit `{ path /bar }`.

Maybe haproxy has already a keyword or configuration for that use case, if so 
I'd love to hear some advices. If not this is a draft of a proposal that I'd 
love to hear from you if it's something doable:

case (acl-name|anonymous-acl) {
  http-request bar # outer acl implicit
  http-request baz if { hdr(x-bar) baz } # outer acl implicitly ANDed here
  case (acl-name|anonymous-acl) {
http-request bazz # both outer acls ANDing here
  }
}

case keyword starts a block, a closing brace alone in the line closes the inner 
block. Indentation doesn't matter at all. haproxy now knows that if the outer 
acl doesn't match, all the inner keywords should be ignored. Does it make sense?

~jm

map_dir() fail when using path as input

[Joao Morais]

Hi list, giving the configuration below, I was expecting "ok1" response instead 
of "ok2". What am I doing wrong?

/tmp/p1

/ ok


h.cfg

defaults
timeout client 1m
timeout server 1m
timeout connect 5s
mode http
listen l1
bind :8000
http-request set-var(req.path) path
http-request set-var(req.res1) path,map_dir(/tmp/p1)
http-request set-var(req.res2) var(req.path),map_dir(/tmp/p1)
http-request return string ok1 content-type text/plain if { var(req.res1) 
-m str ok }
http-request return string ok2 content-type text/plain if { var(req.res2) 
-m str ok }
http-request return string fail content-type text/plain


$ haproxy -v
/ $ haproxy -v
HAProxy version 2.5.5-384c5c5 2022/03/14 - https://haproxy.org/
...
Running on: Linux 5.10.77-flatcar #1 SMP Fri Nov 5 17:49:48 -00 2021 x86_64

$ curl localhost:8000
ok2

~jm

changes in 2.5

[Joao Morais]

Hello list, I have a consumer of the master socket’s `show proc` output and I 
observed that 2.5 changed its lay out, and this change lead me to two doubts:

- Is there a release notes or something with all the backward compatibility 
changes between minor versions? I observed that 2.5 now requires that var() has 
a match type, but maybe there are other changes that I’m missing and I should 
take care;
- Do you suggest me a good way to identify the proxy version so I can use a 
distinct parser for the show proc output? I’m currently using the last field of 
the second line, but might be a better option out there that I’m not aware of.

~jm

Re: Limit requests with peers on 2 independent HAProxies to one backend

[Joao Morais]

> Em 8 de nov. de 2021, à(s) 08:26, Aleksandar Lazic  
> escreveu:
> 
> 
> Hi.
> 
> I have 2 LB's which should limit the connection to one backend.
> 
> I would try to use "conn_cur" in a stick table and share it via peers.
> Have anyone such a solution already in place?

Hi Alex, I’ve already posted another question with a similar config which 
worked like a charm in my tests:

https://www.mail-archive.com/haproxy@formilux.org/msg39753.html

~jm



> That's my assuption for the config.
> 
> ```
> peers be_pixel_peers
>  bind 9123
>  log global
>  localpeer {{ ansible_nodename }}
>  server lb1 lb1.domain.com:1024
>  server lb2 lb2.domain.com:1024
> 
> 
> backend be_pixel_persons
>  log global
> 
>  acl port_pixel dst_port {{ dst_ports["pixel"] }}
>  tcp-request content silent-drop if port_pixel !{ src -f 
> /etc/haproxy/whitelist.acl }
> 
>  option httpchk GET /alive
>  http-check connect ssl
>  timeout check 20s
>  timeout server 300s
> 
>  # limit connection to backend
> 
>  stick-table type ip size 1m expire 10m store conn_cur peers be_pixel_peers
>  http-request deny if { src,table_table_conn_cur(sc_conn_cur) gt 100 }
> 
>  
> 
>  http-request capture req.fhdr(Referer) id 0
>  http-request capture req.fhdr(User-Agent) id 1
>  http-request capture req.hdr(host) id 2
>  http-request capture var(txn.cap_alg_keysize)  id 3
>  http-request capture var(txn.cap_cipher) id 4
>  http-request capture var(txn.cap_protocol) id 5
> 
>  http-response set-header X-Server %s
> 
>  balance roundrobin
> 
>  server pixel_persons1 {{ hosts["pixel_persons1"] }}:8184 resolvers mydns ssl 
> check check-ssl ca-file /etc/haproxy/letsencryptauthorityx3.pem maxconn 2 
> weight 20
>  server pixel_persons2 {{ hosts["pixel_persons2"] }}:8184 resolvers mydns ssl 
> check check-ssl ca-file /etc/haproxy/letsencryptauthorityx3.pem maxconn 2 
> weight 20
>  server pixel_persons3 {{ hosts["pixel_persons3"] }}:8184 resolvers mydns ssl 
> check check-ssl ca-file /etc/haproxy/letsencryptauthorityx3.pem maxconn 8 
> weight 80
> 
> ```
> 
> Regards
> Alex
> 

Add vary header with a new value

[Joao Morais]

Hello list, I need to add a Vary header with Origin value in order to properly 
configure a Cors Allow Origin with dynamic content. This works well provided 
that my backend doesn’t need to configure another value to the Vary header. I 
observed that if I configure set-header, this will overwrite the header 
provided by the app; if I configure add-header, this will create another header 
instead of concatenate the values into a single one, as suggested here[1] and 
here[2].

At this moment these are the best configurations that I managed to build:

A)
  http-response set-header vary %[res.hdr(vary)],origin if { res.hdr(vary) -m 
found }
  http-response set-header vary origin if ! { res.hdr(vary) -m found }

B)
  http-response replace-header vary (.*) \1,origin if { res.hdr(vary) -m found }
  http-response set-header vary origin if ! { res.hdr(vary) -m found }

Option A would remove some previous duplicated vary header, preserving just one 
of them; option B would add origin on all duplicated headers. And all of them 
seems to be more verbose than needed.

It seems that I’m missing some haproxy config here. Is there a better or 
suggested way to add a new value to an existing header without adding a new 
whole header line?

~jm

[1] https://datatracker.ietf.org/doc/html/rfc7230#section-3.2.2
[2] https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Vary

Re: no-stop keyword proposal

[Joao Morais]

> Em 27 de jul. de 2021, à(s) 10:03, William Lallemand  
> escreveu:
> 
> On Tue, Jul 20, 2021 at 12:18:18PM -0300, Joao Morais wrote:
>> 
>> Regarding the use case: I need the ability to reach a stopping, but
>> still running haproxy instance to, at least:
>> [...]
>> 
> 
> This is already possible using the master CLI, it was done to access all
> processes, old and new. It was designed to do that kind of things.

Yes! I discovered that a few days ago!


>> diff --git a/src/cli.c b/src/cli.c
>> index b3132191d..4285e5a72 100644
>> --- a/src/cli.c
>> +++ b/src/cli.c
>> @@ -1841,6 +1841,19 @@ static int bind_parse_level(char **args, int cur_arg, 
>> struct proxy *px, struct b
>>  return 0;
>> }
>> 
>> +/* parse the "no-stop" bind keyword */
>> +static int bind_parse_no_stop(char **args, int cur_arg, struct proxy *px, 
>> struct bind_conf *conf, char **err)
>> +{
>> +struct listener *l;
>> +
>> +list_for_each_entry(l, >listeners, by_bind) {
>> +l->options |= LI_O_NOSTOP;
>> +HA_ATOMIC_INC(_jobs);
>> +}
>> +
>> +return 0;
>> +}
>> +
>> static int bind_parse_severity_output(char **args, int cur_arg, struct proxy 
>> *px, struct bind_conf *conf, char **err)
>> {
>>  if (!*args[cur_arg + 1]) {
>> @@ -2984,6 +2997,7 @@ INITCALL1(STG_REGISTER, cfg_register_keywords, 
>> _kws);
>> static struct bind_kw_list bind_kws = { "STAT", { }, {
>>  { "level", bind_parse_level,1 }, /* set the unix socket admin 
>> level */
>>  { "expose-fd", bind_parse_expose_fd, 1 }, /* set the unix socket expose 
>> fd rights */
>> +{ "no-stop",   bind_parse_no_stop, 0 }, /* flag LI_O_NOSTOP in the 
>> listener options */
>>  { "severity-output", bind_parse_severity_output, 1 }, /* set the 
>> severity output format */
>>  { NULL, NULL, 0 },
>> }};
>> 
>> 
> 
> That's not a good idea in my opinion, it's an internal state that should
> be used only in the case of communication between the master and the
> workers, if we expose this to users we will probably have a lot of
> corner cases to handle.
> This keyword is only meant to say to a worker that it must keep the
> communication with the master even if it's trying to exit, so we could
> do some maintenance or debugging over the master CLI.

I completely understand and thanks for the detailed explanation. Things would 
be easier if all of our deployments use master/worker mode, but some of them 
uses standalone servers. Anyway Willy’s suggestion helped here, I configured 
the stats timeout in line with hard-stop-after and I’m connecting to it before 
issue the reload, this give me a established connection to use while sessions 
are alive.

~jm

Re: no-stop keyword proposal

[Joao Morais]

> Em 20 de jul. de 2021, à(s) 14:17, Willy Tarreau  escreveu:
> 
> Hi Joao,
> 
> On Tue, Jul 20, 2021 at 12:18:18PM -0300, Joao Morais wrote:
>> 
>> Hello list, the diff below is a proposal to add a bind keyword used to flag
>> LI_O_NOSTOP option in the bind's listener.
>> 
>> Regarding the use case: I need the ability to reach a stopping, but still
>> running haproxy instance to, at least: 1) fairly distribute shutdown sessions
>> of long running connections (usually websockets) before hard-stop-after
>> timeouts and kicks all the remaining connections at the same time[1]; 2)
>> collect some relevant metrics from a stopping instance, e.g. current sessions
>> and rps, which would be otherwise lost when these metrics are collected only
>> from the current instance.
> 
> It's a bit confusing for me because it mentions two opposite needs, one
> applies to the listeners (and will thus either prevent the new process
> from binding, or randomly distribute connections between the old and the
> new one), and the other one implies killing random and active connections,
> something we don't do at all.

Hi Willy, I think I wasn’t clear about the needs and the proposal itself. I’ve 
actually two needs regarding instances being stopped (SIGUSR) but holding some 
active connections: 1) collect metrics in order to create real data about these 
connections and new data crossing them - some tcp connections and tunnels 
create the most distortions specially in bytes in/out and concurrent 
connections; 2) allow to distribute some shutdown sessions before hard-stop 
expires.

So here is the proposal, which would allow me to connect to an old instance in 
order to change it’s state or collect it’s data, not limited to shutdown 
sessions or metrics: a way to connect to these old instances, so a stats socket 
alive comes to mind.

You’re correct that reuse the same socket would balance requests between older 
and newer instances, depending on the SO_REUSESOCKET state, but as stated below 
the socket would be changed (unix socket, new path on every reload). This 
wouldn’t be an option to use on a static config of course.


>> Regarding the patch: it's just the changes I needed to make and confirm that
>> it works like I was expecting, provided that the listening socket is changed
>> before reloading haproxy into a new instance. Please let me know if such
>> improvement can be made and also if I'm in the right path.
> 
> That's quite of a concern to me because this means that you'll accumulate
> plenty of old processes, even if they do not have any connection at all
> anymore.

The instance wouldn’t run forever - well, I did that test and inc 
unstoppable_jobs seems to be enough to see the instance shutting down after the 
last connection being closed.


> I think that your various needs would have to be addressed differently
> (the killing of active connections and keeping the old process active).
> For example, if you connect to the old process' CLI it will not quit, as
> this socket counts for one. So maybe as long as you can connect there it
> is enough to keep it alive and monitorable ?

This seems promising, I’ll give this a try. It’s the opposite compared with 
what I’m currently doing, I’ll need to take care with the timeout, but at least 
I have something to start right now. Any advice here is very welcome.


> For the connection shutdown, maybe we could extend "shutdown sessions" to
> take a percentage, and it could apply some randomness over all connections.
> This way you could periodically emit some shutdowns to kill 1% of the
> connections every few seconds until you reach 100%. It's possible that
> for high numbers of long connections, this significantly improves the
> reload. I don't know if we could even easily automate this, but I do
> see some value in it. It could sometimes kill some stats connections
> as well, but with a bit of cheating that could be avoided.

Maybe configure something that works with hard-stop-after? Adding another 
duration or a percentage of the whole hard-stop-after config where the shutdown 
starts - so a hard-stop-after of 30m and the configuration asking HAProxy to 
start the shutdown in the last 10% of the whole hard-stop-after time would 
start to shutdown 1000 session in the last 3 minutes, closing about 5-6 
sessions per second.

-Joao Morais

no-stop keyword proposal

[Joao Morais]

Hello list, the diff below is a proposal to add a bind keyword used to flag 
LI_O_NOSTOP option in the bind’s listener.

Regarding the use case: I need the ability to reach a stopping, but still 
running haproxy instance to, at least: 1) fairly distribute shutdown sessions 
of long running connections (usually websockets) before hard-stop-after 
timeouts and kicks all the remaining connections at the same time[1]; 2) 
collect some relevant metrics from a stopping instance, e.g. current sessions 
and rps, which would be otherwise lost when these metrics are collected only 
from the current instance.

Regarding the patch: it’s just the changes I needed to make and confirm that it 
works like I was expecting, provided that the listening socket is changed 
before reloading haproxy into a new instance. Please let me know if such 
improvement can be made and also if I’m in the right path.

~jm

[1] https://www.mail-archive.com/haproxy@formilux.org/msg40916.html

diff --git a/src/cli.c b/src/cli.c
index b3132191d..4285e5a72 100644
--- a/src/cli.c
+++ b/src/cli.c
@@ -1841,6 +1841,19 @@ static int bind_parse_level(char **args, int cur_arg, 
struct proxy *px, struct b
return 0;
 }

+/* parse the "no-stop" bind keyword */
+static int bind_parse_no_stop(char **args, int cur_arg, struct proxy *px, 
struct bind_conf *conf, char **err)
+{
+   struct listener *l;
+
+   list_for_each_entry(l, >listeners, by_bind) {
+   l->options |= LI_O_NOSTOP;
+   HA_ATOMIC_INC(_jobs);
+   }
+
+   return 0;
+}
+
 static int bind_parse_severity_output(char **args, int cur_arg, struct proxy 
*px, struct bind_conf *conf, char **err)
 {
if (!*args[cur_arg + 1]) {
@@ -2984,6 +2997,7 @@ INITCALL1(STG_REGISTER, cfg_register_keywords, _kws);
 static struct bind_kw_list bind_kws = { "STAT", { }, {
{ "level", bind_parse_level,1 }, /* set the unix socket admin 
level */
{ "expose-fd", bind_parse_expose_fd, 1 }, /* set the unix socket expose 
fd rights */
+   { "no-stop",   bind_parse_no_stop, 0 }, /* flag LI_O_NOSTOP in the 
listener options */
{ "severity-output", bind_parse_severity_output, 1 }, /* set the 
severity output format */
{ NULL, NULL, 0 },
 }};

Re: missing sessions from show sess

[Joao Morais]

> Em 16 de jul. de 2021, à(s) 22:16, Joao Morais  
> escreveu:
> 
> ...
> 
> # show sess
> 
>$ awk '{print $4}' sess |sort |uniq -c
>  1
>  1 fe=GLOBAL
>   1902 fe=_front__tls
>  2 fe=_front_http
> 38 fe=_front_https
> 
> ...
> 
> The third output shows the number of current sessions per frontend, 
> _front__tls seems to be correct, however http and https are apparently 
> missing lots of sessions. Shouldn’t the other frontends have a similar number 
> of sessions compared with their show stat counterpart? Is there a way to find 
> and list the remaining http and https connections?


Now  I think I can answer myself, after a few tests on a local environment, 
maybe this can be useful for someone in the future having the same doubts 
(including me).

_front__tls is a mode tcp proxy, so the number of active sessions, listed by 
show sess, will be the same the number of connections to that frontend.

_front_http and _front_https are mode http proxies, so the number of active 
sessions, listed by show sess, will be the number of running http requests. As 
soon as the http request finishes, the session disappear from the list, event 
though the client is still connected to the frontend - global ConnCurrs counts 
this.

The difference between global ConnCurrs (show info) or scur (show stat) and the 
list above (show sess) is apparently the active connections, counting in show 
info and show stat, without a running http request or a http tunnel (eg 
websocket), so without counting in show sess. If my assumption is correct, "  
4. scur [LFBS]: current sessions" from the management guide could be clarified 
on how it works on mode http frontends.

~jm

missing sessions from show sess

[Joao Morais]

Hi there, I read frontend stats, global info and current sessions from a 
running haproxy, almost at the same time. Here are the outputs:

# show stat -1 1 -1 typed -- all frontends

$ sed -En '/(pxname|scur)/s/.*://p' fronts
_front__tls
1906
_front_http
40
_front_https
1862
stats
0
prometheus
3
healthz
0

# show info

$ grep CurrConns info
CurrConns: 3809

# show sess

$ awk '{print $4}' sess |sort |uniq -c
  1
  1 fe=GLOBAL
   1902 fe=_front__tls
  2 fe=_front_http
 38 fe=_front_https

The first two outputs seem correct: show info says 3809 and all the frontends 
sum 3811.

The third output shows the number of current sessions per frontend, _front__tls 
seems to be correct, however http and https are apparently missing lots of 
sessions. Shouldn’t the other frontends have a similar number of sessions 
compared with their show stat counterpart? Is there a way to find and list the 
remaining http and https connections?

Here is part of the configuration file:

listen _front__tls
mode tcp
bind :443,:::443
... some ssl-passthrough related keywords
server _default_server_https_socket 
unix@/var/run/haproxy/_https_socket.sock send-proxy-v2
frontend _front_http
mode http
bind :80,:::80
...
frontend _front_https
mode http
bind unix@/var/run/haproxy/_https_socket.sock accept-proxy ssl ...
…


~jm

fairly distribute shutdown session a few minutes before hard-stop-after expires

[Joao Morais]

Hello list, we have a HAProxy cluster in front of some chat-like applications. 
This HAProxy cluster is dynamically updated and now and then the instances need 
to be reloaded.

Some of the applications behind this cluster have a few thousand of active 
users and, every time that the old instance’s hard-stop-after expires, closing 
the remaining connections, the applications receive an avalanche of 
reconnections from the clients (usually browsers) almost at the same time.

What I’m planning to do is to monitor every old instance and start a `shutdown 
session` via cli, a few connections at a time, fairly distributing them in the 
last 25% or so of the remaining time. However maybe someone has a better idea 
or even pointing to a configuration that does more or less what I’m trying to 
implement.

~jm

Re: SNI spoofing in HAproxy?

[Joao Morais]

> Em 5 de jul. de 2021, à(s) 09:30, Froehlich, Dominik 
>  escreveu:
> 
> Here is my iteration of your solution:
> 
>  http-request set-var(txn.host) hdr(host),field(1,:)
>  acl ssl_sni_http_host_match ssl_fc_sni,strcmp(txn.host) eq 0
>  http-request deny deny_status 421 if !ssl_sni_http_host_match { 
> ssl_fc_has_sni }

The last line doesn’t make sense to me if you use mTLS. Using this 
configuration you’d let the client choose to not inform the SNI extension and 
use whatever header he want to use. You should either block the request if SNI 
and Host header differs despite the use of the SNI extension (which will lead 
to block due to the match failing) or ignore the SNI at all. The former should 
be used on multi site deployments, the later should be used when you have only 
one single domain per ip+port. You cannot mix both, you cannot optionally use 
the SNI extension and also use mTLS on multi site deployments for security 
reason.

~jm

Re: Inconsistent reading of txn vars from Lua script

[Joao Morais]

> Em 12 de mai. de 2021, à(s) 02:47, Willy Tarreau  escreveu:
> 
> On Tue, May 11, 2021 at 05:41:28PM -0300, Joao Morais wrote:
> 
>> Just to confirm how it works, I created the snippet below:
>> 
>>http-request lua.auth ## assigning txn.core
>>http-request return lf-string %[var(txn.code)] content-type text/plain
>> 
>> It worked since the first run and this is the only place I declared txn.code.
>> Does this mean that a var is created in the following conditions?
>> Any change in the sentences below?
>> 
>> - after the first read from a  Lua script
>> - after the first write from a Lua script provided that ifexists parameter 
>> is set to false
>> - always exists, if used anywhere in the configuration file
> 
> It's not a matter of first or second access. It's that the function
> you used initially resulted in always allocating an entry for the
> variable's name, causing some huge memory usage for those who were
> using them like maps and performing random lookups there. In order
> to avoid this, Tim added an extra argument saying that we're just
> performing an opportunistic lookup and that the variable must not
> be created if it does not exist.

Afaics this is only an option for set_var()? Here is the doc:

TXN.set_var(TXN, var, value[, ifexist])
TXN.get_var(TXN, var)
http://www.arpalert.org/src/haproxy-lua-api/2.3/index.html
(link is broken from haproxy site, is the source versioned
somewhere so I can send a patch?)

So lookups would still create variables, hence the “in the second run it works” 
- my second script (which reads) create the entry and my first script (which 
writes with ifexists true) can find it and update in the second run and beyond.

> Other parts of the code (the native config parts I mean) which use
> variables always result in a creation because these names are static.
> So my understanding is that it can be simplified to this:
>  - a variable declared in the config always exists
>  - a variable accessed from Lua with ifexists set to true will not
>be created but will be found if it exists
>  - a vraiable accessed from Lua with ifexists set to false or not
>present will always be created during the lookup.

Taking the sentence above I’d update this to:

- s/variable accessed/variable updated, set_var(),/
- variable accessed from Lua, get_var(), will always be created

Does this make any sense?

Re: Inconsistent reading of txn vars from Lua script

[Joao Morais]

> Em 10 de mai. de 2021, à(s) 18:04, Willy Tarreau  escreveu:
> 
> On Mon, May 10, 2021 at 10:41:36PM +0200, Willy Tarreau wrote:
>>> core.register_action("auth", { "http-req" }, function(txn)
>>> txn:set_var("txn.code", 401, true)
> 
> So the problem is exactly here and it works as designed. This
> argument "ifexist" was added a year ago to avoid Lua allocating
> random variable names:
> 
>  4e172c93f ("MEDIUM: lua: Add `ifexist` parameter to `set_var`")
> 
> What the "true" argument does here is to refrain from creating
> the variable if it does not exist. After you look it up from the
> service, the variable gets created and it exists, hence why it
> then works next times.
> 
> If you want it to always be created (which I assume you want
> to), just drop this argument or explicitly set it to false.

Thanks Willy for the explanation and sorry about the false alarm, I didn’t see 
the whole picture here.

Just to confirm how it works, I created the snippet below:

http-request lua.auth ## assigning txn.core
http-request return lf-string %[var(txn.code)] content-type text/plain

It worked since the first run and this is the only place I declared txn.code. 
Does this mean that a var is created in the following conditions? Any change in 
the sentences below?

- after the first read from a  Lua script
- after the first write from a Lua script provided that ifexists parameter is 
set to false
- always exists, if used anywhere in the configuration file

Thanks,

Inconsistent reading of txn vars from Lua script

[Joao Morais]

Hello again! Here are the snippets running with 2.4-dev18 - docker image 
haproxy:2.4-dev18-alpine:

$ cat h.cfg
global
  log stdout format raw local0
  lua-load /tmp/h/svc1.lua
  lua-load /tmp/h/svc2.lua
defaults
  timeout server 1m
  timeout client 1m
  timeout connect 5s
  log global
listen l
  mode http
  bind :8000
  option httplog
  http-request lua.auth
  http-request use-service lua.send-failure

$ cat svc1.lua
core.register_action("auth", { "http-req" }, function(txn)
txn:set_var("txn.code", 401, true)
end, 0)

$ cat svc2.lua
core.register_service("send-failure", "http", function(applet)
response = applet:get_var("txn.code")
if response ~= nil then
applet:set_status(response)
else
applet:set_status(403)
end
applet:add_header("Content-Length", 0)
applet:add_header("Content-Type", "text/plain")
applet:start_response()
end)

Now curl’ing the config above:

$ curl -i localhost:8000
HTTP/1.1 403 Forbidden
content-type: text/plain
content-length: 0

$ curl -i localhost:8000
HTTP/1.1 401 Unauthorized
content-type: text/plain
content-length: 0

The first run is always a 403 which means that the reading of the txn.code 
retuned nil, all the next attempts correctly returns 401. Maybe I’m missing 
some kind of initialization here? Otherwise I’m happy to provide this as a 
GitHub issue.

~jm

Read req scoped var from a Lua service

[Joao Morais]

Hello list, following a few questions about Lua and HAProxy vars:

Is there a way to read req scoped vars from a Lua script registered with 
core.register_service()? My attempts so far didn’t succeed, I need to copy the 
value to a txn scoped var before call the service.

Another question, one of these vars has a comma separated list of strings that 
I declare with str(itemOne,itemTwo), but HAProxy complains with:

error detected in proxy 'l' while parsing 'http-request set-var(txn.myvar)' 
rule : fetch method 'str' : expected ')' before ‘,itemTwo)’.

Tested with 2.2 and 2.4. No matter I escape the comma or place inside quotes, 
the error is the same. Is there a way to circumvent this behavior?

Last but not least haproxy.org points to a non existent Lua reference manual, 
maybe /2.0dev/ should be changed to a stable version.

~jm

How to read the state of a mTLS connection

[Joao Morais]

Hi, I started logging mTLS connection failures in order to understand how 
frequently and why they fail for our users. From the collected data I observed 
that about 1% of the client certificates has some kind of issue that neither 
ssl_c_ca_err nor ssl_c_err reports, but it’s reported by ssl_c_verify; about 
0.5% or so has distinct errors between ssl_c_ca_err+ssl_c_err and ssl_c_verify; 
and almost 99% has ssl_c_verify reporting exactly what ssl_c_ca_err and/or 
ssl_c_err reports. Added `ca-ignore-err all` and `crt-ignore-err all` in the 
bind line in order to have a chance to collect these datas.

My doubt is how these samples intersect - if this happens at all? Maybe there 
are other samples out there that should be used and I’m missing them?

~jm

Configure peers on clusters with 20+ instances

[Joao Morais]

Hello list. I'm implementing peers in order to share rps and other metrics 
between all instances of a haproxy cluster, so I have a global view of these 
data. Here is a snippet of my poc which simply does a request count:

global
localpeer h1
...
listen l1
...
http-request track-sc0 int(1) table p/t1
http-request set-var(req.gpc0) sc_inc_gpc0(0)
http-request set-var(req.gpc0) sc_get_gpc0(0,p/t2),add(req.gpc0)
http-request set-var(req.gpc0) sc_get_gpc0(0,p/t3),add(req.gpc0)
http-request return hdr x-out %[var(req.gpc0)]
peers p
bind :9001
log stdout format raw local0
server h1
server h2 127.0.0.1:9002
server h3 127.0.0.1:9003
table t1 type integer size 1 store gpc0
table t2 type integer size 1 store gpc0
table t3 type integer size 1 store gpc0

Our biggest cluster has actually 25 haproxy instances, meaning 25 tables per 
instance, and 25 set-var + add() per request per tracking data. On top of that 
all the 25 instances will share their 25 tables to all of the other 24 
instances. Build and maintain such configuration isn't a problem at all because 
it's automated, but how does it scale? Starting from how much instances should 
I change the approach and try, eg, to elect a controller that receives 
everything from everybody and delivers grouped data? Any advice or best 
practice will be very much appreciated, thanks!

~jm

Re: parsing reload cmd output from master cli

[Joao Morais]

> Em 5 de fev. de 2021, à(s) 04:33, Willy Tarreau  escreveu:
> 
> On Wed, Feb 03, 2021 at 06:32:33PM -0300, Joao Morais wrote:
>> 
>> Hello William, here[1] is some context. I implemented a parsing of the 
>> reload command sent to the master cli, leave it running for a while and got 
>> an index out of bounds (in my code) this week.
>> 
>> I'm using this lay out to parse the reload output:
>> 
>> //   1   3   4   6   
>> 8   8
>> //   
>> 0...|...6...|...2...|...8...|...4...|...0...|...8
>> //
>> //   # 
>> 
>> //   1   master  0   2   
>> 0d00h01m28s 2.2.3-0e58a34
>> //   # workers
>> //   3   worker  1   0   
>> 0d00h00m00s 2.2.3-0e58a34
>> //   # old workers
>> //   2   worker  [was: 1]1   
>> 0d00h00m28s 2.2.3-0e58a34
>> //   # programs
>> //
>> 
>> Apparently I found a line that: starting char isn't '#', have 32 chars or
>> more, have less than 48 chars. Is that even possible?
> 
> I had a quick look at the code producing this output and it writes
> full lines at once, but the second one is a string which is mostly a
> constant ("worker", "master"), or taken from a structure for old
> programs. So I'm seeing two possibilities:
>  - either your parser received an incomplete line due to network buffering
>and decided to parse it before getting the trailing "\n"; this is a
>pretty common issue in line-oriented parsers ported to consume data
>from other processes ;
> 
>  - or the process memory was corrupted so that an old program's "child->id"
>contained some crap and a "\n", which would have resulted in printing
>the PID, this crappy string, and the rest of the line on a second separate
>line.
> 
> The last point seems pretty possible given that your outdated version is
> affected by 123 known bugs, 2 of which could cause memory corruption.

So we probably ended with the first one - currently using 2.2.8, this output 
came from my drafts when I was designing the remote restart. =)


>> How would you expand the lay out above, so I can improve my parser and my 
>> tests? Thanks!
> 
> Please read the chunk_appendf() statements in cli_io_handler_show_proc(),
> they're pretty explicit and you'll get the exact output format for various
> types of dumps. But that's basically what's documented above, it's just
> that it may help you to figure some constants there (or that some fields
> are unsigned for example).

Thank you so much for the insight! This will help us to prepare the code for 
such corner cases.

~jm

Re: A faster way to lookup 5k hosts+paths

[Joao Morais]

> Em 5 de fev. de 2021, à(s) 04:17, Willy Tarreau  escreveu:
> 
> Hi Joao,
> 
> On Tue, Feb 02, 2021 at 09:03:06PM -0300, Joao Morais wrote:
>> 
>> Hello list, I've about 5000 hostnames + path that should be mapped to 3000 
>> backends or so. I'm using map converters and the lay out is something like 
>> this:
>> 
>> /dir/file.map
>>d1.tld/path/sub back1
>>d1.tld/path back2
>>d2.tld/path/sub/other back3
>>d2.tld/path/sub back4
>>d2.tld/path back5
>>d3.tld/path/sub back6
>>d3.tld/path back7
>> 
>> And frontend looks like this:
>> 
>>http-request set-var(req.backend) base,map_beg(/dir/file.map)
>>use_backend %[var(req.backend)] if { var(req.backend) -m found }
>> 
>> The problem here is that the map has about 5k lines and the search isn't
>> indexed. I could eg split this map into smaller ones and create an index of
>> maps stored in another map, but cannot use the result because the file name
>> param in the converter cannot be dynamic. Any other idea to help haproxy find
>> the backend in a faster way?
> 
> Not sure why you're saying it's not indexed. It uses map_beg so it will
> use a prefix tree for the lookup, it should be pretty fast.

Hi Willy, thanks for clarifying this. My question and my concern were based on 
the doc which says "Other keys are stored in lists, so the first matching 
occurrence will be used.”. My understanding were that it wasn’t indexed, but 
instead comparing one item at a time from the top, specially when map_reg 
belongs to other keys as well. So it’s good to know that map_beg (and also 
map_dir?) already does a pretty good job.


> By the way, you could simplify your rules this way:
> 
> use_backend base,map_beg(/dir/file.map)
> 
> Indeed, a use_backend rule which doesn't find the backend will do nothing
> and continue to next rules.

Ah sure, I’m aware of this construction - except the fact that I can safely 
remove the condition, tks - but I’ve actually some other configs between the 
converter and the use_backend. Should’ve used a “…” between both in the 
example. =)

~jm

parsing reload cmd output from master cli

[Joao Morais]

Hello William, here[1] is some context. I implemented a parsing of the reload 
command sent to the master cli, leave it running for a while and got an index 
out of bounds (in my code) this week.

I'm using this lay out to parse the reload output:

//   1   3   4   6  
 8   8
//   
0...|...6...|...2...|...8...|...4...|...0...|...8
//
//   #
 
//   1   master  0   2   
0d00h01m28s 2.2.3-0e58a34
//   # workers
//   3   worker  1   0   
0d00h00m00s 2.2.3-0e58a34
//   # old workers
//   2   worker  [was: 1]1   
0d00h00m28s 2.2.3-0e58a34
//   # programs
//

Apparently I found a line that: starting char isn't '#', have 32 chars or more, 
have less than 48 chars. Is that even possible?

How would you expand the lay out above, so I can improve my parser and my 
tests? Thanks!

~jm

[1] https://www.mail-archive.com/haproxy@formilux.org/msg38415.html

A faster way to lookup 5k hosts+paths

[Joao Morais]

Hello list, I've about 5000 hostnames + path that should be mapped to 3000 
backends or so. I'm using map converters and the lay out is something like this:

/dir/file.map
d1.tld/path/sub back1
d1.tld/path back2
d2.tld/path/sub/other back3
d2.tld/path/sub back4
d2.tld/path back5
d3.tld/path/sub back6
d3.tld/path back7

And frontend looks like this:

http-request set-var(req.backend) base,map_beg(/dir/file.map)
use_backend %[var(req.backend)] if { var(req.backend) -m found }

The problem here is that the map has about 5k lines and the search isn't 
indexed. I could eg split this map into smaller ones and create an index of 
maps stored in another map, but cannot use the result because the file name 
param in the converter cannot be dynamic. Any other idea to help haproxy find 
the backend in a faster way?

~jm

Change precedence using maps and distinct match types

[Joao Morais]

Hello list, I'm configuring a couple of maps used to route requests based on 
hostname and path. The map lay out is pretty much like this:

sub.tld/path1 name_backend1
...

I have some distinct ways to match a request: str, beg, dir, reg, using their 
map converter derivatives.

map_str()
sub.tld/path/exact-match-based name_backend1
...

map_beg()
sub.tld/path/beginning-based name_backend2
...

map_dir()
sub.tld/path/subdir-based name_backend3
...

map_reg()
sub.tld/path/regex-based name_backend4
...


Here is a haproxy frontend snippet:

http-request set-var(req.backend) base,map_str(/dir/to/map_str)
http-request set-var(req.backend) base,map_beg(/dir/to/map_beg) if !{ 
var(req.backend) -m found }
http-request set-var(req.backend) base,map_dir(/dir/to/map_dir) if !{ 
var(req.backend) -m found }
http-request set-var(req.backend) base,map_reg(/dir/to/map_reg) if !{ 
var(req.backend) -m found }

Everything works pretty good provided that paths under the same hostname 
doesn't overlap each other, or if they overlap, they belong to the same 
matching pattern. However we have scenarios like this:

map_reg() -- sub.tld/path/sub[0-9]/?.* name_backend1
map_beg() -- sub.tld/path name_backend2
map_reg() -- sub.tld/p[a-z].* name_backend3

The order of the lines states the precedence - first match should win, but the 
way my config was built doesn't allow to implement this. /path/sub1 will proxy 
to backend2 if map_beg() comes first (backend1 should win), and /path/x will 
proxy to backend3 if map_reg() comes first (backend2 should win). I'm using 
maps due to performance reasons, we've currently about 20k lines of 
hostname+path and about 5k backends.

To the best of my knowledge I'll need to create several maps per match type, 
and reorder them accordingly - but maybe I'm missing something here. Any advice?

~jm

Re: [PATCH] DOC: clarify how to create a fallback crt

[Joao Morais]

> Em 24 de nov de 2020, à(s) 05:47, William Lallemand  
> escreveu:
> 
> Hello Joao,
> 
> On Sat, Nov 21, 2020 at 12:33:38PM -0300, Joao Morais wrote:
>> 
>> It’s indeed rather confusing, sorry about the mess.
>> 
>> Here is a new proposal of the last paragraph, how it sounds? - suggestions 
>> welcome, note that I’m not very familiar with english
>> 
>> 
>> 
>>  The first declared certificate of a bind line is used as the default
>>  certificate, either from crt or crt-list option, which haproxy should use in
>>  the TLS handshake if no other certificate matches. This certificate will 
>> also
>>  be used if the provided SNI matches its CN or SAN, even if a matching SNI
>>  filter is found on any crt-list. The SNI filter !* can be used after the 
>> first
>>  declared certificate to not include its CN and SAN in the SNI tree, so it 
>> will
>>  never match except if no other certificate matches. This way the first
>>  declared certificate act as a fallback.
> 
> It looks good in my opinion, can you make a new patch for it?

Sure! Attached a new patch on top of current master.





0001-DOC-better-describes-how-to-configure-a-fallback-crt.patch
Description: Binary data

Re: [PATCH] DOC: clarify how to create a fallback crt

[Joao Morais]

> Em 21 de nov de 2020, à(s) 12:00, William Lallemand  
> escreveu:
> 
> On Sat, Nov 21, 2020 at 07:48:48AM -0300, Joao Morais wrote:
>> 
>> The attached patch adds some clarification on how one can declare a
>> proper fallback certificate using crt-list. Feel free to ask me to
>> tune verbosity to a higher or lower level.
>> 
> 
> That's actually a bit confusing, because the first line of a crt-list is
> not the default certificate. The default certificate is the first
> certificate declared on a bind line.
> 
> For example:
> 
> bind :443 ssl crt default.pem crt-list list1.crtlist
> bind :443 ssl crt-list list1.crtlist crt-list list2.crtlist
> 
> In the first case, the fallback certificate will be "default.pem", and
> in the second case, it will be the fist line of "list1.crtlist”.

It’s indeed rather confusing, sorry about the mess.

Here is a new proposal of the last paragraph, how it sounds? - suggestions 
welcome, note that I’m not very familiar with english



  The first declared certificate of a bind line is used as the default
  certificate, either from crt or crt-list option, which haproxy should use in
  the TLS handshake if no other certificate matches. This certificate will also
  be used if the provided SNI matches its CN or SAN, even if a matching SNI
  filter is found on any crt-list. The SNI filter !* can be used after the first
  declared certificate to not include its CN and SAN in the SNI tree, so it will
  never match except if no other certificate matches. This way the first
  declared certificate act as a fallback.

[PATCH] DOC: clarify how to create a fallback crt

[Joao Morais]

The attached patch adds some clarification on how one can declare a proper 
fallback certificate using crt-list. Feel free to ask me to tune verbosity to a 
higher or lower level.




0001-DOC-clarify-how-to-create-a-fallback-crt.patch
Description: Binary data

Re: Use default/first crt only if all snifilter fails

[Joao Morais]

> Em 17 de nov de 2020, à(s) 05:28, William Lallemand  
> escreveu:
> 
> You could also do
> 
> /tmp/default.pem !*
> 
> That will ignore the creation of the SNI entries.

Wow thank you so much Willian, as far as I can tell and based on ~5min tests, 
this worked like a charm without any backward compatibility issue.

I consider this a nice addition to the doc that should be merged as far as the 
`!*` snifilter works. I can provide a patch if you consider this a feature 
instead of a hack, let me know if I can help.

~jm

Use default/first crt only if all snifilter fails

[Joao Morais]

Hello list, I have a `crt-list` keyword configuring a list of crt/keys, 
something like this:

/tmp/default.pem
/tmp/a.pema.local
/tmp/b.pemb.local

We consider the first line the fallback certificate - that one that should be 
used if everything else fails.

We've however one situation where default.pem is valid for a.local, and since 
default.pem is the first crt due to its fallback status, default.pem is used in 
the handshake instead of a.pem.

Is there a way to configure a certificate to just act as a fallback crt? Any 
other advice here?

~jm

Re: check successful reload using master cli

[Joao Morais]

> Em 15 de set de 2020, à(s) 12:36, William Lallemand  
> escreveu:
> 
> Oh right... the space in "[was: ]" is troublesome for cutting the string,
> we must remove it.

It's not a problem at all when using chunks of fixed size, even if columns 
differ between them, and the lay out ([was: ...]) remains the same.

> Regarding the size of the fields, 16 bytes should be enough but there is
> the expection of the version field that can be much larger if a dev
> version with the commit ID in the version is used.

If changing the spec, please provide also a way to distinguish the haproxy 
version via master cli which is not in the same `show proc` command, something 
like a `show info` or `version`, or at most in a fixed position at the start of 
`show proc`, so I can use it to version the output parser.

~jm

Re: check successful reload using master cli

[Joao Morais]

> Em 14 de set de 2020, à(s) 19:14, William Lallemand  
> escreveu:
> 
> Hello,
> 
> On Mon, Sep 14, 2020 at 12:09:21PM -0300, Joao Morais wrote:
>> Hello list, I'm working on an automation around haproxy process
>> lifecycle in master-worker mode. It's working nice but I'm not
>> confident that all premisses I used are correct. Please provide some
>> guidance if I did any wrong assumption, RTFM link is welcome as usual.
>> 
> 
> The documentation about the master CLI is probably light about how it's
> implemented so I'll try to be as clear as possible.
> 
Cristal clear, thank you so much. Just a few more doubts below.


>> First of all I figured out that master cli will "connection refused"
>> after a reload cmd during the start of the new worker. I'm using a
>> unix socket but I assume the behaviour would be the same using
>> ip+port.
> 
> Indeed, during a reload all connections to the master CLI are closed
> because the master process is executed again. There is currently no
> mecanism to recover a previous connection from the previous process.
> That's the same with the unix socket or an IP socket.
> 
Is there any chance to be fast enough to connect and receive a response to a 
`show proc` command before the master started the reload process? Sleep 1ms 
maybe? Or when master finishes the `reload` response it’ll immediately shutdown 
its listener?


>> I also observed that the `show proc` output isn't that
>> machine-friendly, anyway it's not a big deal, some regex does the job
>> and I'm assuming the lay out is immutable. Maybe I missed some
>> optional json output in the doc?
>> 
> It is supposed to be splitable easily with cut, but if that does not
> work this is a clearly a bug. 

So may I assume fields have 16 bytes and relative pid is represented as "[was: 
<0-9...>]", otherwise we have a bug?

~jm

check successful reload using master cli

[Joao Morais]

Hello list, I'm working on an automation around haproxy process lifecycle in 
master-worker mode. It's working nice but I'm not confident that all premisses 
I used are correct. Please provide some guidance if I did any wrong assumption, 
RTFM link is welcome as usual.

First of all I figured out that master cli will "connection refused" after a 
reload cmd during the start of the new worker. I'm using a unix socket but I 
assume the behaviour would be the same using ip+port. I'm also using a 
successful connection as the meaning of a finished reload process, failed or 
not. Please let me know if there is a way to be notified, I'm retrying to 
connect within a for-loop and a growing sleep between attempts.

Another assumption I made is that if I don't have any worker in the `# workers` 
list, it means that the last reload failed. I've observed that as soon as the 
master cli accepts connections again, the new worker is already there provided 
that the reload didn't fail.

I also observed that the `show proc` output isn't that machine-friendly, anyway 
it's not a big deal, some regex does the job and I'm assuming the lay out is 
immutable. Maybe I missed some optional json output in the doc?

Thanks in advance, ~jm

Redefine 401 error page

[Joao Morais]

Hello list, the 401 is one of the http status code haproxy generates itself:

https://github.com/haproxy/haproxy/blob/v2.1.0/doc/configuration.txt#L363

This cannot however be overwritten using the errorfile keyword as stated in the 
doc:

https://github.com/haproxy/haproxy/blob/v2.1.0/doc/configuration.txt#L3558

and also testing myself:

[WARNING] 142/002731 (1) : parsing [/tmp/haproxy/h.cfg:9] : status code 401 
not
handled by 'errorfile', error customization will be ignored.

I'm aware that a Lua script can generate a custom page and an arbitrary http 
status code which could work around this:

core.register_service("send-401", "http", function(applet)
send(applet, 401, [[
My custom 401 page
]])
end)

... but is there a way to, instead, customize the output of `http-request auth`?

Config:

$ cat h.cfg
defaults
  timeout client 1s
  timeout server 1s
  timeout connect 1s
  mode http
listen l
  bind :8000
  http-request auth if { always_true }
  errorfile 401 /tmp/haproxy/401.http
  server l 10.0.0.10:8000

Output:

$ curl -i localhost:8000 --user a:b
HTTP/1.1 401 Unauthorized
content-length: 112
cache-control: no-cache
content-type: text/html
www-authenticate: Basic realm=“l"
connection: close

401 Unauthorized
You need a valid user and password to access this content.


~jm

Improve a metric collector

[Joao Morais]

Hello list. I’m improving a metric collector for a haproxy cluster and want to 
confirm if my findings and sentenses below are correct. My main goal using 
these metrics is to know how far from exhaustion my haproxy cluster is.

1. Source of the metric:

I’m parsing `show info` from admin socket and collecting Idle_pct. I’m 
collecting this metric every 500ms because this is the exact time between 
Idle_pct updates. If I’d collect more often this would just be a spend of 
haproxy time for nothing. If I’d collect less often I’d loose information 
leading to an imprecise result. Out of curiosity: I’m converting Idle_pct back 
to time in order to put the amount of processing time in a time series db. This 
allows me to put the metric in distinct resolutions and easily convert back to 
idle/busy pct using `rate()`.

2. Meaning of Idle_pct in a multi-threaded deployment:

If I understood the code correctly, whenever haproxy process my `show info` 
command, a distinct thread might be responsible for that. Every thread has its 
own Idle_pct calc and they are not shared, I cannot eg query the mean Idle_pct 
between all running threads. On one side I’m always looking to a fraction of 
the haproxy process on every collect, on the other side the workload should be 
equally distributed between all threads so the metric should be almost the 
same, with some negligible difference between them.

3. Meaning of the latency collecting `show info`

I’m also using the latency during the collect of the metric to measure how fast 
haproxy is doing its job. If the mean time to collect a response for `show 
info` is growing, the same should be happening with client requests and server 
responses as well, because neither the TCP sockets used for client/server talk 
nor the unix socket for admin queries has a shortcut, all of them are processed 
in the same queue, in the same order the kernel received them.

Are my sentenses correct? Is there anything else I could have a look in order 
to improve my metrics?

Re: ModSecurity testing

[Joao Morais]

> Em 13 de dez de 2019, à(s) 10:09, Christopher Faulet  
> escreveu:
> 
> Le 10/12/2019 à 05:24, Igor Cicimov a écrit :
>> 
>> Testing with Haproxy 2.0.10 but same result with 1.8.23. The versions of 
>> ModSecurity is 2.9.2 and the OWASP rules v3.0.2
>> What am I doing wrong? Can anyone provide a request that should confirm if 
>> the module is working or not from or share the experience from their own 
>> setup?
> 
> Hi Igor,
> 
> First of all, I don't know how the modsecurity agent really work. But I'm 
> surprised to see it returns -101. In the code, -1, 0 or an HTTP status code 
> is expected. And only 0 or the HTTP status code is returned to HAProxy. I 
> don't know if -101 is a valid return value from modsecurity point of view. 
> But it is not from the agent one.
> 
> Then, You don't have an error 403 because the variable txn.modsec.code is 
> negative, so the deny http-request rule is never triggered. So, I guess your 
> error 400 comes from your webserver. You can enabled HTTP log to have more 
> information.
> 
> Finally, I notice some requests to the SPOA agent seems to have failed. The 
> variable is not set (- in the logs). You can try to enable SPOE logs in your 
> SPOE engine configuration. Take a look at the SPOE documentation 
> (doc/SPOE.txt) for more information.


Hi, perhaps this thread helps:

https://www.mail-archive.com/haproxy@formilux.org/msg30061.html

And perhaps this building of ModSecurity SPOA will also help:

https://github.com/jcmoraisjr/modsecurity-spoa/blob/v0.5/rootfs/Dockerfile

~jm

Re: [PATCH] BUG/MINOR: config: Warn cookie domain only if missing embedded dot

[Joao Morais]

Hi Willy,

> Em 30 de out de 2019, à(s) 01:41, Willy Tarreau  escreveu:
> 
> Hi Joao,
> 
> On Tue, Oct 29, 2019 at 09:10:11PM -0300, Joao Morais wrote:
>> 
>> What I need to implement is a way to share the sticky session cookie between
>> two distinct but related domains, say haproxy.org and haproxy.com, without
>> its subdomains. I couldn't do that in a way that worked and without a
>> warning.
> 
> Interesting, that's probably what forced browsers to ignore the leading
> dot when in the early 2000s we started to see host-less site names. Then
> I think we can indeed include your patch but then we also need to change
> the comment mentioning RFC2109 and update it to 6265.

Yup, attached a new patch. Note however I only know two or three words in 
english beyond “hello” and “good morning”, so feel free to update subject to a 
proper one and comment to what is really happening and what the spec is really 
forbiding or not. No need to be reviewed. =)

Thanks!

~jm





0001-BUG-MINOR-config-Update-cookie-domain-warn-to-RFC626.patch
Description: Binary data

Re: [PATCH] BUG/MINOR: config: Warn cookie domain only if missing embedded dot

[Joao Morais]

Hi Willy,

> Em 29 de out de 2019, à(s) 04:27, Willy Tarreau  escreveu:
> 
> No, please look at the RFC again, it's very precise on this :
> https://tools.ietf.org/html/rfc2109

Thanks for taking the time to review my patch.

In fact I read RFC 6265 which doesn’t take the leading dot as mandatory. 6265 
obsoletes 2965, which obsoletes 2109.

What I need to implement is a way to share the sticky session cookie between 
two distinct but related domains, say haproxy.org and haproxy.com, without its 
subdomains. I couldn’t do that in a way that worked and without a warning.

~jm

[PATCH] BUG/MINOR: config: Warn cookie domain only if missing embedded dot

[Joao Morais]

Hi list, the attached patch fixes a warn message if the domain option, from 
cookie keyword, configures a domain without starting with a dot.

~jm





0001-BUG-MINOR-config-Warn-cookie-domain-only-if-missing-.patch
Description: Binary data

Re: segmentation fault on 1.8.21

[Joao Morais]

> Em 23 de ago de 2019, à(s) 08:16, Willy Tarreau  escreveu:
> 
> On Fri, Aug 23, 2019 at 11:47:46AM +0200, Willy Tarreau wrote:
>> In the mean time you can apply the patch above. It will reject the
>> first hunk but the second one applies and will address the issue.
> 
> I've now backported all the pending patches for 1.8. You can git-pull
> it if it's easier for you.

Hi Willy, current 1.8 code (commit dcb8c97) fixes the issue we found on spoe, 
thanks!

> Thank you for this report

Thank you all for the great job on haproxy.

~jm

segmentation fault on 1.8.21

[Joao Morais]

Hi list, I can reproduce a segmentation fault on HAProxy 1.8.21. No problem 
with 1.8.20, 1.9.10 or 2.0.5. Is there anything else I can provide or test on 
my environment?

--

haproxy.cfg:
...
frontend f
mode http
...
filter spoe engine modsecurity config /etc/haproxy/spoe-modsecurity.conf
...
backend spoe-modsecurity
mode tcp
timeout connect 5s
timeout server  5s
server modsec-spoa0 192.168.100.99:12345
...

--

spoe-modsecurity.conf:
[modsecurity]
spoe-agent modsecurity-agent
messages check-request
option   var-prefix  modsec
timeout  hello   11s
timeout  idle28s
timeout  processing  39s
use-backend  spoe-modsecurity
spoe-message check-request
args   unique-id method path query req.ver req.hdrs_bin req.body_size 
req.body
event  on-frontend-http-request

--

Backtrace:

Program received signal SIGSEGV, Segmentation fault.
0x55f747657c3c in spoe_encode_message (s=s@entry=0x55f7485bf320,
ctx=ctx@entry=0x55f7485bf740, msg=msg@entry=0x55f7484ba680,
dir=dir@entry=0, buf=buf@entry=0x7ffe4fb78088,
end=end@entry=0x55f7485d7910 "") at src/flt_spoe.c:2190
2190src/flt_spoe.c: No such file or directory.
(gdb) bt
#0  0x55f747657c3c in spoe_encode_message (s=s@entry=0x55f7485bf320,
ctx=ctx@entry=0x55f7485bf740, msg=msg@entry=0x55f7484ba680,
dir=dir@entry=0, buf=buf@entry=0x7ffe4fb78088,
end=end@entry=0x55f7485d7910 "") at src/flt_spoe.c:2190
#1  0x55f7476585f5 in spoe_encode_messages (type=, dir=0,
messages=0x55f7484ba4a0, ctx=0x55f7485bf740, s=0x55f7485bf320)
at src/flt_spoe.c:2233
#2  spoe_process_messages (s=0x55f7485bf320, ctx=0x55f7485bf740,
messages=, dir=dir@entry=0, type=type@entry=1)
at src/flt_spoe.c:2616
#3  0x55f74765a145 in spoe_process_event (ev=,
ctx=, s=) at src/flt_spoe.c:2703
#4  spoe_chn_pre_analyze (s=, filter=,
chn=0x55f7485bf330, an_bit=) at src/flt_spoe.c:3129
#5  0x55f7476c1595 in flt_pre_analyze (s=s@entry=0x55f7485bf320,
chn=chn@entry=0x55f7485bf330, an_bit=an_bit@entry=16) at src/filters.c:805
#6  0x55f74764dfd5 in process_stream (t=t@entry=0x55f7485bf680)
at src/stream.c:1906
#7  0x55f7476d5844 in process_runnable_tasks () at src/task.c:229
#8  0x55f747683214 in run_poll_loop () at src/haproxy.c:2419
#9  run_thread_poll_loop (data=) at src/haproxy.c:2489
#10 0x55f7475e261c in main (argc=, argv=)
at src/haproxy.c:3092

Send http 413 response

[Joao Morais]

Hello list. I'm trying to send a HTTP 413 to the user based on the 
hdr(Content-Length). What I've tried so far:

1. Create a http413 backend only with `errorfile 400` + `http-request 
deny_status 400`. In the frontend, configure a `use_backend http413 if 
`. This is my current approach but it is wasting some time in the 
frontend for every single request of every single backend - we have about 1000 
backends and only about 10% need to check Content Length - btw distinct content 
lengths.

2. Use the `errorfile 400` approach in the same backend that does the load 
balance. This doesn't sound good because I'm overwriting some internal response 
code and its payload. Btw, what if I need another 2, 3, 10 http response codes?

3. Use some creativity eg: errorfile 413 + deny_status 413; use_backend inside 
another backend; what more? The later doesn't make sense but the former is a 
pitty that isn't supported.

Is there another way to deny a http request with a custom status and html 
content that I’m missing? Thanks!

~jm

Re: Host header and sni extension differ

[Joao Morais]

Hi Willy,

> Em 17 de mai de 2019, à(s) 04:03, Willy Tarreau  escreveu:
> 
> Hi Jarno,
> 
> On Thu, May 16, 2019 at 06:49:56PM +0300, Jarno Huuskonen wrote:
>> Do the myapp.io and anotherapp.com share same certificate (ie.
>> certificate has both myapp.io and anotherapp.com SAN) ?
>> 
>> AFAIK browser can reuse the same tls connection if the certificate
>> covers both names.
> 
> Absolutely, I've already read about this though I don't know the
> implementations details. Similar concepts have been discussed quite
> a bit on the HTTP WG, though I don't undertand the details of each
> variation. The main thing is that sometimes the browser will consider
> that the connection is safe to be used for another domain name because
> the first one is considered authoritative on it. I'm not sure whether
> it only learns this from the cert or also from some response headers
> though. This is also why I always say that routing on SNI is wrong
> and that only the Host header is relevant.

Everything was working without a single problem because we in fact route our 
requests based on Host header. The problem started when we need validate some 
requests based on client certs and, randomly, the DN and SHA1 headers wasn’t 
provided.


>> When the host/sni differ do you have an earlier
>> connection (for example from same ip/port) using matching sni/host in your
>> logs ?
> 
> Normally there should indeed be one.

Yep, I can now confirm this one. A few minutes before the divergent Host x SNI, 
the user made some requests to the domain of the “wrong” SNI.

~jm

Re: Host header and sni extension differ

[Joao Morais]

Hey guys,

> Em 16 de mai de 2019, à(s) 15:05, Tim Düsterhus  escreveu:
> 
> Am 16.05.19 um 17:49 schrieb Jarno Huuskonen:
>> Do the myapp.io and anotherapp.com share same certificate (ie.
>> certificate has both myapp.io and anotherapp.com SAN) ?
>> 
>> AFAIK browser can reuse the same tls connection if the certificate
>> covers both names. When the host/sni differ do you have an earlier
>> connection (for example from same ip/port) using matching sni/host in your
>> logs ?

I can now confirm that 100% of the distinct SNIs has certificates whose SAN 
also matches the Host hostname. Thank you so much for bringing some light to 
this problem! Time to test some patches in the config file.

~jm

Host header and sni extension differ

[Joao Morais]

Hi list! The symptom is as follow: when logging Host: header I receive 
`myapp.io` while in the same request the sni extension says `anotherapp.com`.

This happens in a very few requests (about 0.5%) but this is enough to make 
some noise - regarding server certificate used in the handshake, and also the 
ca-file used in handshakes with client certs. When they differ, the header is 
right and the sni is wrong.

I can confirm that every "myapp.io" or "anotherapp.com" resolves to the same 
haproxy cluster. I can also confirm that all agents are browsers (Chrome and 
Firefox) running in Linux and, based on the "myapp.io" and "anotherapp.com" 
samples I saw together in the logs, the user is using both applications at the 
same time, probably from the same instance of the browser.

Is there something the browser and/or HAProxy is or isn't doing here in order 
to mess host header and sni?

Some config snippets, any change suggestion? (besides upgrade HAProxy)

HAProxy 1.8.19

global
daemon
nbthread 3
cpu-map auto:1/1-3 0-2
stats socket /var/run/haproxy-stats.sock level admin expose-fd listeners
maxconn 12000
hard-stop-after 6m
log 127.0.0.1:5140 format rfc5424 local0
log-tag ingress
lua-load /usr/local/etc/haproxy/lua/send-response.lua
lua-load /usr/local/etc/haproxy/lua/auth-request.lua
ssl-dh-param-file /ingress-controller/ssl/dhparam.pem
ssl-default-bind-ciphers ...
ssl-default-bind-options ssl-max-ver TLSv1.2 ssl-min-ver TLSv1.0
tune.bufsize 65536
defaults
log global
option redispatch
option dontlognull
option http-server-close
option http-keep-alive
timeout http-request5s
timeout connect 5s
timeout client  300s
timeout client-fin  70s
timeout queue   5s
timeout server  300s
timeout server-fin  70s
timeout tunnel  1h
timeout http-keep-alive 70s

~jm

DoS in h2 - from f5.com

[Joao Morais]

Hi list, do you know if HAProxy wasn't mentioned here[1] because it isn't 
vulnerable (1.8 and 1.9) or because it wasn't tested?

~jm

[1] 
https://www.f5.com/labs/articles/threat-intelligence/denial-of-service-vulnerabilities-discovered-in-http-2

Re: Tune HAProxy in front of a large k8s cluster

[Joao Morais]

> Em 20 de fev de 2019, à(s) 02:51, Igor Cicimov 
>  escreveu:
> 
> 
> On Wed, 20 Feb 2019 3:39 am Joao Morais  Hi Willy,
> 
> > Em 19 de fev de 2019, à(s) 01:55, Willy Tarreau  escreveu:
> > 
> > use_backend foo if { var(req.host) ssl:www.example.com }
> > 
> This is a nice trick that I’m planning to use with dynamic use_backend. I 
> need to concat host (sometimes ssl_fc_sni) and path. The question is: how do 
> I concatenate two strings?
> 
> Something like this:
> http-request set-header X-Concat %[req.fhdr(Authorization),word(3,.)]_%[src]
> 
Hi Igor, this is almost the same workaround I’m using - the difference is that 
I’m moving the result to a req.var and removing the header after that. 
Wondering if 1.8 has a better option 

~jm

Re: Tune HAProxy in front of a large k8s cluster

[Joao Morais]

> Em 20 de fev de 2019, à(s) 03:30, Baptiste  escreveu:
> 
> Hi Joao,
> 
> I do have a question for you about your ingress controller design and the 
> "chained" frontends, summarized below:
> * The first frontend is on tcp mode binding :443, inspecting sni and doing a 
> triage;
>There is also a ssl-passthrough config - from the triage frontend straight 
> to a tcp backend.
> * The second frontend is binding a unix socket with ca-file (tls 
> authentication);
> * The last frontend is binding another unix socket, doing ssl-offload but 
> without ca-file.
> 
> What feature is missing in HAProxy to allow switching these 3 frontends into 
> a single one?
> I understand that the ability to do ssl deciphering and ssl passthrough on a 
> single bind line is one of them. Is there anything else we could improve?
> I wonder if crt-list would be useful in your case: 
> https://cbonte.github.io/haproxy-dconv/1.9/configuration.html#5.1-crt-list
> 
Hi Baptiste, I’m changing the approach of the frontend creation - if the user 
configuration just need one, this one will listen :443 without need to chain 
another one. Regarding switch to more frontends - or at least more bind lines 
in the same frontend - and creating the mode-tcp one, here are the current 
rules:

* conflict on timeout client - and perhaps on other frontend configs - distinct 
frontends will be created to each one
* if one really want to use a certificate that doesn’t match its domain - 
crt-list sounds to solve this
* tls auth (bind with ca-file) and no tls auth - I don’t want to mix then in 
the same frontend because of security - tls auth use sni, no tls auth use host 
header
* ssl-passthrough as you have mentioned

~jm

Re: %[] in use-server directives

[Joao Morais]

> Em 19 de fev de 2019, à(s) 17:51, Bruno Henc  escreveu:
> 
> On 2/19/19 9:45 PM, Joao Morais wrote:
>> 
>>> Em 19 de fev de 2019, à(s) 05:57, Willy Tarreau  escreveu:
>>> 
>>> In the past it was not possible
>>> to dynamically create servers
>> I think I misunderstood something, but... how do one dynamically create a 
>> new server?
>> 
> The following links should be able to help you out:
> 
> https://www.haproxy.com/blog/dynamic-configuration-haproxy-runtime-api/#dynamically-scaling-backend-servers
> 
> https://www.haproxy.com/blog/dynamic-scaling-for-microservices-with-runtime-api/#runtime-api
> 
> You might need to build a development version of HAProxy to take advantage of 
> the latest features.
> 
> 
> Let me know if you get stuck.
> 
Hi Bruno, thanks! Updating servers via api I’m currently using. From Willy "in 
the past it was not possible to dynamically create servers" - so now I’m 
wondering if there is a way or future plan to create a new server on an 
existing backend and, even better, also dynamically create a new backend and 
its servers.

~jm

Re: %[] in use-server directives

[Joao Morais]

> Em 19 de fev de 2019, à(s) 05:57, Willy Tarreau  escreveu:
> 
> In the past it was not possible
> to dynamically create servers

I think I misunderstood something, but... how do one dynamically create a new 
server?

Re: Tune HAProxy in front of a large k8s cluster

[Joao Morais]

Hi Willy,

> Em 19 de fev de 2019, à(s) 01:55, Willy Tarreau  escreveu:
> 
> use_backend foo if { var(req.host) ssl:www.example.com }
> 
This is a nice trick that I’m planning to use with dynamic use_backend. I need 
to concat host (sometimes ssl_fc_sni) and path. The question is: how do I 
concatenate two strings? Apparently there isn’t a concat converter and 
http-request set-var() doesn’t support custom-log like expressions. There is a 
usecase where I need to concatenate ssl_fc_sni and path before search in the 
map.


> At this point I think that such heavy configs reach their limits and
> that the only right solution is the dynamic use_backend (possibly with
> a map).
> 
Thanks for the detailed review! I’m going to the map route.


>> There are also a lot of other backends and
>> servers with health check enabled every 2s consuming some cpu and network.
> 
> For this if you have many times the same server you can use the "track"
> directive, and only enable checks on a subset of servers and have all
> other track them. Typically you'd have a dummy backend dedicated to
> checks, and checks disabled in all other backends, replaced with track.
> 
I’d say that currently about 98% are unique servers, but this is indeed a nice 
implementation to the configuration builder.


>> Note also that I needed to add -no-pie otherwise gprof output was empty --
>> sounds a gcc issue. Let me know if this is good enough.
> 
> Yes that's fine and the output was perfectly exploitable.
> 
Great!

One final note - sorry about the flood yesterday. I can say with about 90% sure 
I sent only one message =)

~jm

Re: Tune HAProxy in front of a large k8s cluster

[Joao Morais]

> Em 16 de fev de 2019, à(s) 03:16, Willy Tarreau  escreveu:
> 
> If you have some time to run some extra tests, it would be nice to rebuild
> haproxy with "ARCH_FLAGS=-pg", run it again, stop it using kill -USR1 (not
> ctrl-C), and run "gprof haproxy gmon.out". It will show the number of calls
> to each function and a rough approximation of the time spent there. We may
> find a huge number of calls to the culprit and possibly we could improve
> it.

Hi Willy, sure. Attached gprof output of a 300rps request during 20s, total of 
6000 requests. The request is empty (has only trivial headers) and output body 
has only 4 bytes. There are 12 servers on the backend which wait 200ms before 
sending its response. There are also a lot of other backends and servers with 
health check enabled every 2s consuming some cpu and network. Note also that I 
needed to add -no-pie otherwise gprof output was empty — sounds a gcc issue. 
Let me know if this is good enough.

~jm



gprof.txt.gz
Description: GNU Zip compressed data

Re: Tune HAProxy in front of a large k8s cluster

[Joao Morais]

> Em 16 de fev de 2019, à(s) 03:16, Willy Tarreau  escreveu:
> 
> If you have some time to run some extra tests, it would be nice to rebuild
> haproxy with "ARCH_FLAGS=-pg", run it again, stop it using kill -USR1 (not
> ctrl-C), and run "gprof haproxy gmon.out". It will show the number of calls
> to each function and a rough approximation of the time spent there. We may
> find a huge number of calls to the culprit and possibly we could improve
> it.

Hi Willy, sure. Attached gprof output of a 300rps request during 20s, total of 
6000 requests. The request is empty (has only trivial headers) and output body 
has only 4 bytes. There are 12 servers on the backend which wait 200ms before 
sending its response. There are also a lot of other backends and servers with 
health check enabled every 2s consuming some cpu and network. Note also that I 
needed to add -no-pie otherwise gprof output was empty — sounds a gcc issue. 
Let me know if this is good enough.

~jm



gprof.txt.gz
Description: GNU Zip compressed data

Re: Tune HAProxy in front of a large k8s cluster

[Joao Morais]

> Em 16 de fev de 2019, à(s) 03:16, Willy Tarreau  escreveu:
> 
> If you have some time to run some extra tests, it would be nice to rebuild
> haproxy with "ARCH_FLAGS=-pg", run it again, stop it using kill -USR1 (not
> ctrl-C), and run "gprof haproxy gmon.out". It will show the number of calls
> to each function and a rough approximation of the time spent there. We may
> find a huge number of calls to the culprit and possibly we could improve
> it.

Hi Willy, sure. Attached gprof output of a 300rps request during 20s, total of 
6000 requests. The request is empty (has only trivial headers) and output body 
has only 4 bytes. There are 12 servers on the backend which wait 200ms before 
sending its response. There are also a lot of other backends and servers with 
health check enabled every 2s consuming some cpu and network. Note also that I 
needed to add -no-pie otherwise gprof output was empty — sounds a gcc issue. 
Let me know if this is good enough.

~jm



gprof.txt.gz
Description: GNU Zip compressed data

Re: Tune HAProxy in front of a large k8s cluster

[Joao Morais]

> Em 16 de fev de 2019, à(s) 03:16, Willy Tarreau  escreveu:
> 
> If you have some time to run some extra tests, it would be nice to rebuild
> haproxy with "ARCH_FLAGS=-pg", run it again, stop it using kill -USR1 (not
> ctrl-C), and run "gprof haproxy gmon.out". It will show the number of calls
> to each function and a rough approximation of the time spent there. We may
> find a huge number of calls to the culprit and possibly we could improve
> it.

Hi Willy, sure. Attached gprof output of a 300rps request during 20s, total of 
6000 requests. The request is empty (has only trivial headers) and output body 
has only 4 bytes. There are 12 servers on the backend which wait 200ms before 
sending its response. There are also a lot of other backends and servers with 
health check enabled every 2s consuming some cpu and network. Note also that I 
needed to add -no-pie otherwise gprof output was empty — sounds a gcc issue. 
Let me know if this is good enough.

~jm



gprof.txt.gz
Description: GNU Zip compressed data

Re: Tune HAProxy in front of a large k8s cluster

[Joao Morais]

> Em 16 de fev de 2019, à(s) 03:16, Willy Tarreau  escreveu:
> 
> If you have some time to run some extra tests, it would be nice to rebuild
> haproxy with "ARCH_FLAGS=-pg", run it again, stop it using kill -USR1 (not
> ctrl-C), and run "gprof haproxy gmon.out". It will show the number of calls
> to each function and a rough approximation of the time spent there. We may
> find a huge number of calls to the culprit and possibly we could improve
> it.

Hi Willy, sure. Attached gprof output of a 300rps request during 20s, total of 
6000 requests. The request is empty (has only trivial headers) and output body 
has only 4 bytes. There are 12 servers on the backend which wait 200ms before 
sending its response. There are also a lot of other backends and servers with 
health check enabled every 2s consuming some cpu and network. Note also that I 
needed to add -no-pie otherwise gprof output was empty — sounds a gcc issue. 
Let me know if this is good enough.

~jm



gprof.txt.gz
Description: GNU Zip compressed data

Re: Tune HAProxy in front of a large k8s cluster

[Joao Morais]

> Em 16 de fev de 2019, à(s) 03:16, Willy Tarreau  escreveu:
> 
> If you have some time to run some extra tests, it would be nice to rebuild
> haproxy with "ARCH_FLAGS=-pg", run it again, stop it using kill -USR1 (not
> ctrl-C), and run "gprof haproxy gmon.out". It will show the number of calls
> to each function and a rough approximation of the time spent there. We may
> find a huge number of calls to the culprit and possibly we could improve
> it.

Hi Willy, sure. Attached gprof output of a 300rps request during 20s, total of 
6000 requests. The request is empty (has only trivial headers) and output body 
has only 4 bytes. There are 12 servers on the backend which wait 200ms before 
sending its response. There are also a lot of other backends and servers with 
health check enabled every 2s consuming some cpu and network. Note also that I 
needed to add -no-pie otherwise gprof output was empty — sounds a gcc issue. 
Let me know if this is good enough.

~jm



gprof.txt.gz
Description: GNU Zip compressed data

Re: Tune HAProxy in front of a large k8s cluster

[Joao Morais]

> Em 16 de fev de 2019, à(s) 03:16, Willy Tarreau  escreveu:
> 
> If you have some time to run some extra tests, it would be nice to rebuild
> haproxy with "ARCH_FLAGS=-pg", run it again, stop it using kill -USR1 (not
> ctrl-C), and run "gprof haproxy gmon.out". It will show the number of calls
> to each function and a rough approximation of the time spent there. We may
> find a huge number of calls to the culprit and possibly we could improve
> it.

Hi Willy, sure. Attached gprof output of a 300rps request during 20s, total of 
6000 requests. The request is empty (has only trivial headers) and output body 
has only 4 bytes. There are 12 servers on the backend which wait 200ms before 
sending its response. There are also a lot of other backends and servers with 
health check enabled every 2s consuming some cpu and network. Note also that I 
needed to add -no-pie otherwise gprof output was empty — sounds a gcc issue. 
Let me know if this is good enough.

~jm



gprof.txt.gz
Description: GNU Zip compressed data

Re: Tune HAProxy in front of a large k8s cluster

[Joao Morais]

> Em 15 de fev de 2019, à(s) 19:22, Aleksandar Lazic  
> escreveu:
> 
> Am 15.02.2019 um 22:11 schrieb Joao Morais:
>> 
>> Hey Aleks, this made my day. Thank you for remember me that map exist and a
>> big thank you to The Author of map, map_beg and map_reg converters! Time to
>> achieve a 5 digits rps.
> 
> Wow you mean from 600 rps to 6 rps "just" to switch to map.
> That's amazing 8-O, especially it was just a wild guess ;-)
> 
This is just theory - and 5 digits starts on 1 =) . The problem is the 
"  if   " multiplied by 3000 or so. Moving 
everything to the backend or a map is a really big step forward. I suspect my 
frontend will have about 10 lines instead of 4000.


> Would you like to share your config now?
> 
I’d share but it has some sensitive data. But trust in me, this is as simple as 
one tcp frontend on :443 with about 100 use_backend -> one http frontend on an 
unix socket with about 3000 " if” with a few regex -> the backend.

I'll let you know as soon as I've updated the config and ran the same tests 
again ;)

~jm

Re: Tune HAProxy in front of a large k8s cluster

[Joao Morais]

> Em 15 de fev de 2019, à(s) 08:43, Aleksandar Lazic  
> escreveu:
> 
> Hi Joao.
> 
> Am 15.02.2019 um 11:15 schrieb Joao Morais:
>> 
>> Hi Aleks, sure. Regarding the config, it has currently about 4k lines only 
>> in the largest frontend because of the number of hostnames and paths being 
>> supported. About 98% is acl declarations, http-request, reqrep, redirect 
>> scheme, use_backend. Most of them I'll move to the backend and this will 
>> already improve performance. The question is: what about the 2200+ 
>> `use_backend` - is there anything else that could be done?
> 
> As I don't know the config, even a snippet could help, let me suggest you to 
> try
> to use a map for lookup for the backends.
> 
> https://www.haproxy.com/blog/introduction-to-haproxy-maps/
> 
Hey Aleks, this made my day. Thank you for remember me that map exist and a big 
thank you to The Author of map, map_beg and map_reg converters! Time to achieve 
a 5 digits rps.


> Do you use DNS resolving for the hostnames?
> 
Nops. My guess so far is the size of the frontend and my tests have confirmed 
this.


>> / # haproxy -vv
>> HA-Proxy version 1.8.17 2019/01/08
> 
> Event it's not critical, it would be nice when you can try 1.8.19 or better
> 1.9.4 ;-)
> 
Moving to 1.8.19 soon and waiting 1.9.5 =)

~jm

Re: HAProxy in front of Docker Enterprise problem

[Joao Morais]

> Em 12 de fev de 2019, à(s) 21:21, Norman Branitsky 
>  escreveu:
>  
> Do I have to make HAProxy listen on 8443 and just do a tcp frontend/backend 
> for the Manager nodes?

You can bind on another port, you can also bind on another IP address (change 
*:443 to some.ip.addr:443). But if you want or you need to share the same IP 
and port, a possible configuration is to create a tcp mode frontend which 
inspect sni extension and make a triage: manager hostname? Use a tcp mode 
backend and the manager nodes as servers - no data would be changed. This blog 
post[1] is of some help. In the triage, if the request isn't to a maanger node, 
use another tcp backend whose only server is a unix socket. Use also 
send-proxy-v2 in the server declaration. Create another http mode frontend, 
binding that unix socket and accept-proxy keyword to do the ssl offload of your 
worker nodes. hth.

[1] 
https://www.haproxy.com/blog/enhanced-ssl-load-balancing-with-server-name-indication-sni-tls-extension/

Re: Tune HAProxy in front of a large k8s cluster

[Joao Morais]

> Em 15 de fev de 2019, à(s) 07:44, Aleksandar Lazic  
> escreveu:
> 
> Hi Joao.
> 
> Am 15.02.2019 um 10:21 schrieb Joao Morais:
>> 
>> Hi list, I'm tuning some HAProxy instances in front of a large kubernetes
>> cluster. The config has about 500 hostnames (a la apache/nginx virtual
>> hosts), 3 frontends, 1500 backends and 4000 servers. The first frontend is on
>> tcp mode binding :443, inspecting sni and doing a triage; the second frontend
>> is binding a unix socket with ca-file (tls authentication); the last frontend
>> is binding another unix socket, doing ssl-offload but without ca-file. This
>> last one has about 80% of the hostnames. There is also a ssl-passthrough
>> config - from the triage frontend straight to a tcp backend.
> 
> Please can you tell us which haproxy you use and show us the config, thanks.

Hi Aleks, sure. Regarding the config, it has currently about 4k lines only in 
the largest frontend because of the number of hostnames and paths being 
supported. About 98% is acl declarations, http-request, reqrep, redirect 
scheme, use_backend. Most of them I'll move to the backend and this will 
already improve performance. The question is: what about the 2200+ 
`use_backend` - is there anything else that could be done?

/ # haproxy -vv
HA-Proxy version 1.8.17 2019/01/08
Copyright 2000-2019 Willy Tarreau 

Build options :
  TARGET  = linux2628
  CPU = generic
  CC  = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv 
-Wno-null-dereference -Wno-unused-label
  OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.0.2q  20 Nov 2018
Running on OpenSSL version : OpenSSL 1.0.2q  20 Nov 2018
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.5
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT 
IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.42 2018-03-20
Running on PCRE version : 8.42 2018-03-20
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), 
raw-deflate("deflate"), gzip("gzip")
Built with network namespace support.

Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace

Tune HAProxy in front of a large k8s cluster

[Joao Morais]

Hi list, I'm tuning some HAProxy instances in front of a large kubernetes 
cluster. The config has about 500 hostnames (a la apache/nginx virtual hosts), 
3 frontends, 1500 backends and 4000 servers. The first frontend is on tcp mode 
binding :443, inspecting sni and doing a triage; the second frontend is binding 
a unix socket with ca-file (tls authentication); the last frontend is binding 
another unix socket, doing ssl-offload but without ca-file. This last one has 
about 80% of the hostnames. There is also a ssl-passthrough config - from the 
triage frontend straight to a tcp backend.

I'm observing some latency on moderate loads (200+ rps per instance) - on my 
tests, the p95 was about 25ms only in the proxy, and the major issue is that I 
cannot have a throughput above 600 rps. This latency moves easily from 25ms on 
p95 to 1s or more on p50 with 700+ rps. The problem is of course the big amount 
of rules in the frontend: haproxy need to check every single bit of 
configuration for every single host and every single path. Moving the testing 
hostname to a dedicated frontend with only its own rules give me with about 5ms 
of p95 latency and more than 5000 rps.

These are my ideas so far regarding tune such configuration:

* Move all possible rules to the backend. Some txn vars should be created in 
order to be inspected there. This will of course help but there is still a lot 
of `use_backend if  ` that cannot be removed, I think, 
which are being evaluated on every single request despite the hostname that I'm 
really interested. There are some hostnames without path acl, but there are 
also hostnames with 10+ different paths and its 10+ `use_backend`.

* Create some more frontends and unix sockets with at most 50 hostnames or so. 
Pros: after the triage, each frontend will have the `use_backend if` of only 
another 49 hostnames. Cons: if some client doesn't send the sni extension, the 
right frontend couldn't be found.

* Perhaps there is a hidden `if  do  done` that I'm 
missing which would improve performance, since I can help HAProxy to process 
only the keywords I'm really interested in that request.

* Nbthreads was already tested, I'm using 3 that has the best performance on a 
8 cores VM. 4+ threads doesn’t scale. Nbprocs will also be used, I'm tuning a 
per process configuration now.

Is there any other approach I'm missing? Every single milisecond will help.

Re: Use acl on spoe events

[Joao Morais]

> Em 27 de mai de 2018, à(s) 14:51, Joao Morais  
> escreveu:
> 
>> Em 27 de mai de 2018, à(s) 12:02, Daniel Corbett  
>> escreveu:
>> 
>> Hello Joao,
>> 
>> On 05/26/2018 05:54 PM, Joao Morais wrote:
>>> 
>>> There is no difference if I use acl like the example above, or use the `if 
>>> {...}` syntax or remove the acl at all, my modsecurity agent always receive 
>>> a new connection despite of the host I’m using.
>>> 
>>> Is there a way to use a spoe filter only if some l7 conditions match?
>> 
>> This appears to have been fixed in 1.9-dev with this commit:
>> 
>> https://git.haproxy.org/?p=haproxy.git;a=commitdiff;h=333694d7715952a9610a3e6f00807eaf5edd209a;hp=336d3ef0e77192582c98b3c578927a529ceadd9b
> 
> Hi Daniel, I can confirm this fixes my 1.8.9 deployment, thanks!
> 
> Is there any plan to backport to 1.8?

..

~jm

Right way to seamless reload a multi process cfg

[Joao Morais]

Hi, taking this hypothetical cfg:

===
global
  daemon
  nbproc 3
  stats socket unix@/tmp/haproxy1.sock expose-fd listeners process 1
  stats socket unix@/tmp/haproxy2.sock expose-fd listeners process 2
  stats socket unix@/tmp/haproxy3.sock expose-fd listeners process 3
listen f1
  bind :80 process 1
  server s1 192.168.99.1:8000 check
listen f2
  bind :8080 process 2-3
  server s2 192.168.99.1:8000 check
===

What's the right or the better way to seamless reload (-x) this multi process 
configuration? Multi -x command-line parameter doesn't seem to work, and use 
one or the other socket will apparently leave some listening sockets without 
being reused.

~jm

Re: Use acl on spoe events

[Joao Morais]

> Em 27 de mai de 2018, à(s) 12:02, Daniel Corbett <dcorb...@haproxy.com> 
> escreveu:
> 
> Hello Joao,
> 
> On 05/26/2018 05:54 PM, Joao Morais wrote:
>> 
>> There is no difference if I use acl like the example above, or use the `if 
>> {...}` syntax or remove the acl at all, my modsecurity agent always receive 
>> a new connection despite of the host I’m using.
>> 
>> Is there a way to use a spoe filter only if some l7 conditions match?
> 
> This appears to have been fixed in 1.9-dev with this commit:
> 
> https://git.haproxy.org/?p=haproxy.git;a=commitdiff;h=333694d7715952a9610a3e6f00807eaf5edd209a;hp=336d3ef0e77192582c98b3c578927a529ceadd9b

Hi Daniel, I can confirm this fixes my 1.8.9 deployment, thanks!

Is there any plan to backport to 1.8?

> 
> Thanks,
> -- Daniel

~jm

Use acl on spoe events

[Joao Morais]

Hi list, I’m trying to filter spoe events using acl, no success atm.

This is the relevant part of my configuration:

=== /etc/haproxy/haproxy.cfg ===

frontend bar
...
filter spoe engine modsecurity config /etc/haproxy/spoe-modsecurity.conf
http-request deny if { var(txn.modsec.code) -m int gt 0 }
...
backend spoe-modsecurity
mode tcp
server modsec-spoa1 127.0.0.1:12345

=== /etc/haproxy/spoe-modsecurity.conf

[modsecurity]
spoe-agent modsecurity-agent
messages check-request
option   var-prefix  modsec
timeout  hello   100ms
timeout  idle30s
timeout  processing  1s
use-backend  spoe-modsecurity
spoe-message check-request
aclhost_my.domain req.hdr(host) my.domain
args   unique-id method path query req.ver req.hdrs_bin req.body_size 
req.body
event  on-frontend-http-request if host_my.domain

There is no difference if I use acl like the example above, or use the `if 
{...}` syntax or remove the acl at all, my modsecurity agent always receive a 
new connection despite of the host I’m using.

Is there a way to use a spoe filter only if some l7 conditions match?

~jm

SPOE and modsecurity contrib

[Joao Morais]

Hi list, I'm playing with SPOE and modsecurity contrib from HAProxy 1.8.9. I've 
a couple of doubts and issues that I'll describe just below my config and some 
loggings:

= haproxy.conf
listen my-front
log 127.0.0.1:514 format rfc5424 local0
timeout client 5s
timeout connect 5s
timeout server 5s
mode http
bind :8080
log-format "The txn.modsec.code is: %[var(txn.modsec.code)]"
filter spoe engine modsecurity config /tmp/haproxy/spoe-modsecurity.conf
tcp-request content reject if { var(txn.modsec.code) -m int gt 0 }
server local 192.168.95.102:8000
backend spoe-modsecurity
mode tcp
timeout connect 5s
timeout server  3m
server iprep1 127.0.0.1:12345
-

= spoe-modsecurity.conf
[modsecurity]
spoe-agent modsecurity-agent
messagescheck-request
option  var-prefix  modsec
timeout hello   100ms
timeout idle30s
timeout processing  1s
use-backend spoe-modsecurity
spoe-message check-request
args unique-id method path query req.ver req.hdrs_bin req.body_size req.body
event on-frontend-http-request
-

modsecurity SPOA is logging whenever I make a request to HAProxy so apparently 
it's working as expected:

=log 1 -- curl -I localhost:8080?ref=abc
1526747591.637257 [01] Process SPOE Message 'check-request'
1526747591.638333 [01] Encode Agent ACK frame
1526747591.638867 [01] STREAM-ID=0 - FRAME-ID=1
1526747591.638887 [01] Add action : set variable code=4294967195
-

=log 2 -- curl -I 127.0.0.1:8080?ref=abc
1526741757.670146 [01] Process SPOE Message 'check-request'
1526741757.671956 [00] [client 127.0.0.1] ModSecurity: Warning. Pattern match 
"^[\\d.:]+$" at REQUEST_HEADERS:Host. [file 
"/owasp-modsecurity-crs-3.0.2/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"] 
[line "810"] [id "920350"] [rev "2"] [msg "Host header is a numeric IP 
address"] [data "127.0.0.1:8080"] [severity "WARNING"] [ver "OWASP_CRS/3.0.0"] 
[maturity "9"] [accuracy "9"] [tag "application-multi"] [tag "language-multi"] 
[tag "platform-multi"] [tag "attack-protocol"] [tag 
"OWASP_CRS/PROTOCOL_VIOLATION/IP_HOST"] [tag "WASCTC/WASC-21"] [tag 
"OWASP_TOP_10/A7"] [tag "PCI/6.5.10"] [hostname "8ae61a5f0c2e"] [uri 
"http://127.0.0.1:8080/;] [unique_id ""]
1526741757.672306 [01] Encode Agent ACK frame
1526741757.672333 [01] STREAM-ID=12 - FRAME-ID=1
1526741757.672351 [01] Add action : set variable code=4294967195
-

=log 3 -- curl -I 127.0.0.1:8080?ref=/../etc/passwd
1526741728.786700 [00] [client 127.0.0.1] ModSecurity: Warning. Matched phrase 
"etc/passwd" at ARGS:ref. [file 
"/owasp-modsecurity-crs-3.0.2/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf"] 
[line "448"] [id "932160"] [rev "1"] [msg "Remote Command Execution: Unix Shell 
Code Found"] [data "Matched Data: etc/passwd found within ARGS:ref: 
/etc/passwd"] [severity "CRITICAL"] [ver "OWASP_CRS/3.0.0"] [maturity "1"] 
[accuracy "8"] [tag "application-multi"] [tag "language-shell"] [tag 
"platform-unix"] [tag "attack-rce"] [tag 
"OWASP_CRS/WEB_ATTACK/COMMAND_INJECTION"] [tag "WASCTC/WASC-31"] [tag 
"OWASP_TOP_10/A1"] [tag "PCI/6.5.2"] [hostname "8ae61a5f0c2e"] [uri 
"http://127.0.0.1:8080/;] [unique_id ""]
...
1526741728.787814 [01] Add action : set variable code=4294967195
-

Running modsecurity with: -d -n 1 -f /tmp/haproxy/modsec.conf

My environment is a `haproxy:alpine` container with all the stuff needed to 
compile both modsecurity and it's SPOA contrib. The doc used was 
`haproxy-1.8.9/contrib/modsecurity/README`.

Now a couple of doubts:

* If I `curl -v 127.0.0.1:12345` I can see a connection being opened on 
modsecurity and `curl` waits for something it could understand. If I ctrl+C 
`curl`, modsecurity starts an endless loop and use almost 100% of one CPU. I'm 
aware modsecurity SPOA speaks SPOP instead of HTTP, but perhaps something could 
be improved since this could also happen eg on a network failure.

* The `set variable code=` on logs has always the same value `4294967195` 
despite what modsecurity found.

* A `log-format "%[var(txn.modsec.code)]"` logs always the same value `-101` 
despite what modsecurity found.

* Perhaps a specific HAProxy doubt here: `tcp-request content reject` never 
rejects a request even changing `gt` to `lt` or `eq`. Changing the acl to 
`!{...}` the connection is always rejected despite of the operator.

~jm

Re: [ANNOUNCE] haproxy-1.8.3

[Joao Morais]

> Em 30 de dez de 2017, à(s) 15:32, Willy Tarreau  escreveu:
> 
>  - implemented the graceful shutdown on HTTP/2 connections during a reload
>so that we can inform the client we're going to close, encouraging the
>client to switch to a new connection. This avoids connections from
>lasting forever after reloads on H2. I also noticed that it allows the
>process to be replaced faster.

Hi Willy, is there something like this on 1.7.x? This would be useful on eg 
websockets.

~jm

Re: Client cert verification on some paths

[Joao Morais]

> Em 2 de dez de 2017, à(s) 08:47, Aleksandar Lazic <al-hapr...@none.at> 
> escreveu:
> 
> Von: "Joao Morais" <l...@joaomorais.com.br> gesendet: 02.12.2017 00:53:33
> 
>> Hi, I have some apps that need to mimic an Apache httpd behavior on client 
>> certificate verification: require certificate only on some paths.
>> 
>> Apache does this implementing SSL renegotiation as briefly explained here[1].
>> 
>> Of couse I can `mode tcp` proxy to an Apache instance to do that for me but 
>> my topology would be simplified if I could implement SSL renegotiation on 
>> HAProxy as soon as I can fetch the path sample.
>> 
>> Is there a way to accomplish this without using Apache httpd?
> You can use the following line to full fill your request, untested.
> 
>  bind :443 ssl ca-file "${PATH_TO_CAFILE}" crl-file "${PATH_TO_CRLFILE}" 
> verify "${VERIFY_MODE}"
> 
> http://cbonte.github.io/haproxy-dconv/1.7/configuration.html#5.1
> http://cbonte.github.io/haproxy-dconv/1.7/configuration.html#5.1-ca-file
> http://cbonte.github.io/haproxy-dconv/1.7/configuration.html#5.1-crl-file
> http://cbonte.github.io/haproxy-dconv/1.7/configuration.html#5.1-verify
> 
> You can add the following header to see if the client was successful verified.
> 
> http-request set-header X-SSL-Client-Verify %[ssl_c_verify]
> 
> http://cbonte.github.io/haproxy-dconv/1.7/configuration.html#4.2-http-request
> 
> When you start the haproxy with the environment variables PATH_TO_CAFILE and 
> PATH_TO_CRLFILE set to your paths and VERIFY_MODE=optional can you test if 
> the verification works.
> 
> http://cbonte.github.io/haproxy-dconv/1.7/configuration.html#2.3

Thanks for the detailed explanation.

This is actually very close to my current setup and I'm looking for a way to 
avoid ask the certificate from the user on a browser if he doesn't request the 
/path that require the certificate. HAProxy has a lot of L5/6 fetch samples, 
and with some unknown (by me) keyword, perhaps I could implement a SSL 
renegotiation (or something like that) just like Apache httpd already implement.

Just to name an example: HAProxy doesn't have native support for configuration 
of a http response which explains to the user he need to provide a certificate 
(on one page) - and signed by a known CA (on another page), but I got it 
working using verify optional and fetching the right L5 samples. The actual 
configuration however is far beyond my knowledge in such a way that I simply 
cannot say this is even possible.

> I strongly suggest to go through the manual several times due to the fact 
> that it's worth and you learn a lot about haproxy ;-)

Sure, the link to the doc is already on my favourites =)

~jm

Client cert verification on some paths

[Joao Morais]

Hi, I have some apps that need to mimic an Apache httpd behavior on client 
certificate verification: require certificate only on some paths.

Apache does this implementing SSL renegotiation as briefly explained here[1].

Of couse I can `mode tcp` proxy to an Apache instance to do that for me but my 
topology would be simplified if I could implement SSL renegotiation on HAProxy 
as soon as I can fetch the path sample.

Is there a way to accomplish this without using Apache httpd?

~jm

[1] http://httpd.apache.org/docs/2.4/mod/mod_ssl.html#sslverifyclient

What can cause RST from HAProxy 1.7.9

[Joao Morais]

Hi, HAProxy 1.7.9 is being used to route traffic to a Kubernetes cluster on 
AWS. It was observed periodic spikes of RST from HAProxy on active connections. 
Full details in the following issue from GitHub: 
https://github.com/jcmoraisjr/haproxy-ingress/issues/77 . In which 
circumstances HAProxy send RST? Any help would be very much appreciated.

~jm

[docs] about master-worker mode

[Joao Morais]

Hi, from nbproc doc[1]: "This requires the 'daemon' mode”, but this is also the 
way to start more than one worker on master-worker mode, right?

Still on the same doc: "USING MULTIPLE PROCESSES IS HARDER TO DEBUG AND IS 
REALLY DISCOURAGED”, is this still valid on master-worker? Both "harder to 
debug" and "is really discouraged" parts.

From the blog post[2]: "Start the haproxy process with the command line 
argument -W <# of processes>" - is this <# of procs> correct? Couldn’t use this 
on the command line.

~jm

[1] http://cbonte.github.io/haproxy-dconv/1.8/configuration.html#3.1-nbproc
[2] https://www.haproxy.com/blog/whats-new-haproxy-1-8/

Fetch DN according to RFC 2253

[Joao Morais]

Hi list, is there a way to choose between pre and pos RFC 2253[1] format of DN 
from a client cert? Here are nginx[2] and Apache[3] docs about the subject.

[1] https://tools.ietf.org/html/rfc2253
[2] http://nginx.org/en/docs/http/ngx_http_ssl_module.html#var_ssl_client_s_dn
[3] http://httpd.apache.org/docs/2.4/mod/mod_ssl.html#ssloptions (look for 
LegacyDNStringFormat option)

Re: Deny with 413 request too large

[Joao Morais]

> Em 17 de mai de 2017, à(s) 19:34, Bryan Talbot <bryan.tal...@playnext.com> 
> escreveu:
> 
> 
>> On May 15, 2017, at May 15, 6:35 PM, Joao Morais <l...@joaomorais.com.br> 
>> wrote:
>> 
>>   errorfile 413 /usr/local/etc/haproxy/errors/413.http
>>   http-request deny deny_status 413 if { req.body_size gt 10485760 }
>> 
>> ... HAProxy complains with:
>> 
>>   [WARNING] 135/001448 (27) : parsing [/etc/haproxy/haproxy.cfg:15] : status 
>> code 413 not handled by 'errorfile', error customization will be ignored.
>>   [WARNING] 135/001448 (27) : parsing [/etc/haproxy/haproxy.cfg:89] : status 
>> code 413 not handled, using default code 403.
>> 
>> How should I configure HAProxy in order to deny with 413?
> 
> 
> You’ve already found it. AFAIK, that’s the only way.


=(

In my understanding I should only use a 400badreq.http like message on an 
errorfile 400 config line, otherwise if HAProxy need to issue a 400 status 
code, my 413 status code would be issued instead.

Is this a valid feature request or there are technical reasons why this has 
been done that way?

Hints are welcome.

~jm

Deny with 413 request too large

[Joao Morais]

Hello list, I need to configure HAProxy to deny the request with 413 based on 
the value of the content-length header. This is my actual configuration:

errorfile 400 /usr/local/etc/haproxy/errors/413.http
http-request deny deny_status 400 if { req.body_size gt 10485760 }

This is working but sounds a hacky workaround since I’m using another status 
code. If I try to use:

errorfile 413 /usr/local/etc/haproxy/errors/413.http
http-request deny deny_status 413 if { req.body_size gt 10485760 }

... HAProxy complains with:

[WARNING] 135/001448 (27) : parsing [/etc/haproxy/haproxy.cfg:15] : status 
code 413 not handled by 'errorfile', error customization will be ignored.
[WARNING] 135/001448 (27) : parsing [/etc/haproxy/haproxy.cfg:89] : status 
code 413 not handled, using default code 403.

How should I configure HAProxy in order to deny with 413?

~jm