Re: Performing URL rewriting and re-resolving new URL using configured DNS

2020-09-11 Thread francis Lavalliere
Hello,

Haproxy is a great tool indeed. And as long as you are able to add some dynamic 
template generation of haproxy config based on your own business logic I 
believe you should be good.

You could leverage consul / consul-template and have the haproxy regenerates 
bases on your business logic.

We do use that combination and it works quite well for us.

Regards
Francis L.

Téléchargez Outlook pour Android


From: Yehuda Cohen 
Sent: Friday, September 11, 2020 11:32:51 AM
To: haproxy@formilux.org 
Subject: Performing URL rewriting and re-resolving new URL using configured DNS


Hi all,



Hoping you can help me with this query. I have a selection of micro-backends 
deployed per customer as containers on Amazon ECS. These customers each have a 
public hostname they use to access their respective micro-backend.



Currently we are using Amazon’s ALB to dispatch to these services, but each 
load balancer listener has a maximum of 100 rules that can dispatch traffic. As 
such we are managing multiple load balancers, and this something I’d rather not 
do. We are expecting to onboard thousands of customers and are in the process 
of automating the onboarding workflow.



One solution I thought of was to enable service discovery for each of these 
micro-backends to create an internal SRV record with a domain name that can be 
inferred from the public hostname. For example: 
https://customer-a.mypublicdomain.com to http://customer-a.myinternal.domain.



I’d then like to point each public hostname to a single reverse proxy 
(preferably configured with HAProxy). The reverse proxy would then forward the 
request from the public endpoint to the internal domain name by rewriting the 
Host header replacing mypublicdomain.com with myinternal.domain. It would then 
resolve the SRV record for myinternal.domain using a resolver DNS and forward 
the traffic to the internal micro-backend.



In this configuration, I’m looking to have HAProxy rewrite the url and then 
reresolve the rewritten url with a provided DNS rather than setting up a 
thousand backends and having to provide server configuration.



Is HAProxy a good tool for this use case? If so can anyone point me in a 
direction that I might use to configure this. So far every HAProxy 
configuration manual I’ve found that uses service discovery does so requires a 
backend block per service name. In this case micro-backends are added and 
deleted with very high frequency and I’d rather not modify HAProxy config files 
that frequently.



All the best,

Yehuda


Re: [PATCH] MINOR: Add either(,) converter

2020-09-11 Thread Willy Tarreau
On Fri, Sep 11, 2020 at 05:19:26PM +0200, Tim Düsterhus, WoltLab GmbH wrote:
> Fun. I didn't receive your reply on company mail. I only got it from the
> list using my personal subscription. I hope this message threads properly.

Yep it does.

> Muscle memory is too strong :-/ I even used search and replace to adjust
> the name in all files, except configuration.txt. I even had to fix the
> 'either' within the GPL license comment.
> 
> Willy, find a patch attached. I've put Miroslav as 'Reported-by'.

Thanks, now merged (just after the dev4 release).

Willy



[ANNOUNCE] haproxy-2.3-dev4

2020-09-11 Thread Willy Tarreau
Hi,

HAProxy 2.3-dev4 was released on 2020/09/11. It added 89 new commits
after version 2.3-dev3.

Some might have noticed that we missed the previous dev released two weeks
ago. The fact is that I've been trying hard to attack a deeply rooted old
crap that's been there for more than a decade, and couldn't yet manage to
win the battle after 4 attacks over the last 3 weeks. But I still have yet
another plan. In short, we've been accumulating hacks in the address
management code that's used everywhere an address is parsed, and these
hacks consist in using dummy address families to represent some variants
like socket pairs, external FDs, UDP, QUIC etc. And this has precisely
become a massive obstacle to the rework of listeners that is essential to
get QUIC eventually integrated. This has diverted me long enough to miss
the previous releases. But I never give up and here's a pause for a new
release (without this code yet).

Some of the required code to rework the listeners was merged, including
the one that makes sure we start the listeners in one place (previously
they were started twice, once by scanning the proxies, and once by scanning
the protocols). Despite the main patch being tagged MAJOR, I can't see any
side effect it could have since all listeners were registered in the
protocol lists. But if you discover that some odd service doesn't start
anymore or fails to reload since dev4, please report it.

I've also added the minimally needed changes to let haproxy be built with
the TCC compiler. That's very convenient during development or to quickly
test if a patch broke something, as it builds the whole project in 0.5s
instead of 17s on my machine. Threads are not supported however, but it's
convenient to quickly test potential breakage with various option
combinations.

Tim's "iif()" converter was just merged (a few minutes before the release,
with the trivial doc typo that was reported a few minutes after :-)). This
will likely simplify quite some configs.

There was the usual batch of deinit() cleanups.

One possible user-visible change is that we'll now hard-error on truncted
lines in the config file. It used to appear as a warning in 2.2 and now
it's an error. This never happens, unless you accidently truncated your
file and don't want it to run this way at all! Another visible change is
that configs with duplicate cache section names are now rejected (again,
this must never happen except by accident).

Shimi Gersner added support for SAN extension and certificate chaining when
generating certs on the fly, as by default the emitted certificate didn't
contain the whole CA chain.

The pathq/set-pathq/replace-pathq sample fetch and actions that were merged
into 2.2 were integrated into this version. And the rest is essentially bug
fixes.

Ah, last point, I added a "Work in progress" page in the wiki to list known
long-term developments that are being worked on. I'm currently only aware
of QUIC by Fred, but if others are silently working on certain features that
will take time to get in shape for a merge and they want to add a link there
to avoid effort duplication, they're welcome to do so. Maybe the page is
poorly named, just rename it if you have a better proposal :-)

Please find the usual URLs below :
   Site index   : http://www.haproxy.org/
   Discourse: http://discourse.haproxy.org/
   Slack channel: https://slack.haproxy.org/
   Issue tracker: https://github.com/haproxy/haproxy/issues
   Wiki : https://github.com/haproxy/wiki/wiki
   Sources  : http://www.haproxy.org/download/2.3/src/
   Git repository   : http://git.haproxy.org/git/haproxy.git/
   Git Web browsing : http://git.haproxy.org/?p=haproxy.git
   Changelog: http://www.haproxy.org/download/2.3/src/CHANGELOG
   Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/

Willy
---
Complete changelog :
Bertrand Jacquin (1):
  MINOR: contrib/spoa-server: allow MAX_FRAME_SIZE override

Christopher Faulet (13):
  BUG/MEDIUM: http-ana: Don't wait to send 1xx responses received from 
servers
  MINOR: http-htx: Add an option to eval query-string when the path is 
replaced
  BUG/MINOR: http-rules: Replace path and query-string in "replace-path" 
action
  MINOR: http-htx: Handle an optional reason when replacing the response 
status
  Revert "BUG/MINOR: http-rules: Replace path and query-string in 
"replace-path" action"
  BUG/MEDIUM: doc: Fix replace-path action description
  MINOR: http-rules: Add set-pathq and replace-pathq actions
  MINOR: http-fetch: Add pathq sample fetch
  REGTEST: Add a test for request path manipulations, with and without the 
QS
  BUG/MEDIUM: dns: Don't store additional records in a linked-list
  BUG/MEDIUM: dns: Be sure to renew IP address for already known servers
  MINOR: server: Improve log message sent when server address is updated
  BUG/MEDIUM: pattern: Renew the pattern expression revision when it is 
pruned


Performing URL rewriting and re-resolving new URL using configured DNS

2020-09-11 Thread Yehuda Cohen
Hi all,

Hoping you can help me with this query. I have a selection of micro-backends 
deployed per customer as containers on Amazon ECS. These customers each have a 
public hostname they use to access their respective micro-backend.

Currently we are using Amazon’s ALB to dispatch to these services, but each 
load balancer listener has a maximum of 100 rules that can dispatch traffic. As 
such we are managing multiple load balancers, and this something I’d rather not 
do. We are expecting to onboard thousands of customers and are in the process 
of automating the onboarding workflow.

One solution I thought of was to enable service discovery for each of these 
micro-backends to create an internal SRV record with a domain name that can be 
inferred from the public hostname. For example: 
https://customer-a.mypublicdomain.com to http://customer-a.myinternal.domain.

I’d then like to point each public hostname to a single reverse proxy 
(preferably configured with HAProxy). The reverse proxy would then forward the 
request from the public endpoint to the internal domain name by rewriting the 
Host header replacing mypublicdomain.com with myinternal.domain. It would then 
resolve the SRV record for myinternal.domain using a resolver DNS and forward 
the traffic to the internal micro-backend.

In this configuration, I’m looking to have HAProxy rewrite the url and then 
reresolve the rewritten url with a provided DNS rather than setting up a 
thousand backends and having to provide server configuration.

Is HAProxy a good tool for this use case? If so can anyone point me in a 
direction that I might use to configure this. So far every HAProxy 
configuration manual I’ve found that uses service discovery does so requires a 
backend block per service name. In this case micro-backends are added and 
deleted with very high frequency and I’d rather not modify HAProxy config files 
that frequently.

All the best,
Yehuda


Re: [PATCH] MINOR: Add either(,) converter

2020-09-11 Thread Tim Düsterhus , WoltLab GmbH
Miroslav,

Am 11.09.20 um 17:10 schrieb Miroslav Zagorac:
> there is a small typo in the patch, if says 'iff' instead of 'iif':
> 
> ---
> +  Example:
> +    http-request set-header x-forwarded-proto %[ssl_fc,iff(https,http)]
> ---
> 

Fun. I didn't receive your reply on company mail. I only got it from the
list using my personal subscription. I hope this message threads properly.

Muscle memory is too strong :-/ I even used search and replace to adjust
the name in all files, except configuration.txt. I even had to fix the
'either' within the GPL license comment.

Willy, find a patch attached. I've put Miroslav as 'Reported-by'.

Best regards
Tim Düsterhus
Developer WoltLab GmbH

-- 

WoltLab GmbH
Nedlitzer Str. 27B
14469 Potsdam

Tel.: +49 331 96784338

duester...@woltlab.com
www.woltlab.com

Managing director:
Marcel Werk

AG Potsdam HRB 26795 P
>From e9ea0196ab20d8cd92ba8a9651e0da7e9575f9ae Mon Sep 17 00:00:00 2001
From: Tim Duesterhus 
Date: Fri, 11 Sep 2020 17:13:12 +0200
Subject: [PATCH] DOC: Fix typo in iif() example
To: haproxy@formilux.org
Cc: w...@1wt.eu

It should read 'iif', not 'iff'.

Reported-by: Miroslav Zagorac 
---
 doc/configuration.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/doc/configuration.txt b/doc/configuration.txt
index 0e34f620d..075d3d5c8 100644
--- a/doc/configuration.txt
+++ b/doc/configuration.txt
@@ -15155,7 +15155,7 @@ iif(,)
   string otherwise.
 
   Example:
-http-request set-header x-forwarded-proto %[ssl_fc,iff(https,http)]
+http-request set-header x-forwarded-proto %[ssl_fc,iif(https,http)]
 
 in_table()
   Uses the string representation of the input sample to perform a look up in
-- 
2.28.0



Re: [PATCH] MINOR: Add either(,) converter

2020-09-11 Thread Miroslav Zagorac

Hello all,

there is a small typo in the patch, if says 'iff' instead of 'iif':

---
+  Example:
+http-request set-header x-forwarded-proto %[ssl_fc,iff(https,http)]
---

--
Zaga

What can change the nature of a man?



Re: [PATCH] MINOR: Add either(,) converter

2020-09-11 Thread Willy Tarreau
On Fri, Sep 11, 2020 at 04:55:45PM +0200, Tim Düsterhus, WoltLab GmbH wrote:
> I consider 'iif' a bit obscure. It easily looks like a typo. Similar to
> 'iff' for 'if and only if' which tends to generate a number of questions
> as well.

I agree but others possibly know it and we should not consider our own
case as representative, so let's see.

> But I've certainly seen 'iif' being used before and I don't strongly
> care for the naming. I'm happy with anything.
> 
> I've attached an updated patch and leave it up to you whether you want
> to merge it and whether you'd like to wait for more opinions or not :-)

I've just merged it so that it gets better exposure, that's the best way
to receive criticisms and proposals for another name :-)

Thanks!
Willy



Re: [PATCH] MINOR: Add either(,) converter

2020-09-11 Thread Tim Düsterhus , WoltLab GmbH
Willy,

Am 11.09.20 um 16:46 schrieb Willy Tarreau:
> First, I really like the feature, that's a great idea.

:-)

>>> - choice (my initial choice)
>>> - ifor / if_or
>>> - ifelse / if_else
>>> - iftrue (with the  argument being optional)
>>
>> Maybe something like this would be appropriate (IIF)?
> 
> I was thinking about "select" which I've seldom seen, I never heard about
> "iif" previously but it has the benefit of being quite short (convenient
> for long expressions) and already in use elsewhere. And it also has the
> benefit that someone searching for "if" would easily stumble on it.
> 
> I'm fine with the patch BTW, so if everyone agrees on "iif", let's suggest
> we go with it ?
> 

I consider 'iif' a bit obscure. It easily looks like a typo. Similar to
'iff' for 'if and only if' which tends to generate a number of questions
as well.

But I've certainly seen 'iif' being used before and I don't strongly
care for the naming. I'm happy with anything.

I've attached an updated patch and leave it up to you whether you want
to merge it and whether you'd like to wait for more opinions or not :-)

Best regards
Tim Düsterhus
Developer WoltLab GmbH

-- 

WoltLab GmbH
Nedlitzer Str. 27B
14469 Potsdam

Tel.: +49 331 96784338

duester...@woltlab.com
www.woltlab.com

Managing director:
Marcel Werk

AG Potsdam HRB 26795 P
>From 9022a2f0b3e2ab449debfe6db919312aa97a5262 Mon Sep 17 00:00:00 2001
From: Tim Duesterhus 
Date: Fri, 11 Sep 2020 14:25:23 +0200
Subject: [PATCH] MINOR: Add iif(,) converter
To: haproxy@formilux.org
Cc: w...@1wt.eu

iif() takes a boolean as input and returns one of the two argument
strings depending on whether the boolean is true.

This converter most likely is most useful to return the proper scheme
depending on the value returned by the `ssl_fc` fetch, e.g. for use within
the `x-forwarded-proto` request header.

However it can also be useful for use within a template that is sent to
the client using `http-request return` with a `lf-file`. It allows the
administrator to implement a simple condition, without needing to prefill
variables within the regular configuration using `http-request
set-var(req.foo)`.
---
 doc/configuration.txt   |  7 ++
 reg-tests/converter/iif.vtc | 45 +
 src/sample.c| 22 ++
 3 files changed, 74 insertions(+)
 create mode 100644 reg-tests/converter/iif.vtc

diff --git a/doc/configuration.txt b/doc/configuration.txt
index c1f6f8219..dd8d5889a 100644
--- a/doc/configuration.txt
+++ b/doc/configuration.txt
@@ -15150,6 +15150,13 @@ http_date([])
   microseconds since epoch. Offset is assumed to have the same unit as
   input timestamp.
 
+iif(,)
+  Returns the  string if the input value is true. Returns the 
+  string otherwise.
+
+  Example:
+http-request set-header x-forwarded-proto %[ssl_fc,iff(https,http)]
+
 in_table()
   Uses the string representation of the input sample to perform a look up in
   the specified table. If the key is not found in the table, a boolean false
diff --git a/reg-tests/converter/iif.vtc b/reg-tests/converter/iif.vtc
new file mode 100644
index 0..360b63f96
--- /dev/null
+++ b/reg-tests/converter/iif.vtc
@@ -0,0 +1,45 @@
+varnishtest "iif converter Test"
+
+feature ignore_unknown_macro
+
+server s1 {
+	rxreq
+	txresp
+} -repeat 3 -start
+
+haproxy h1 -conf {
+defaults
+	mode http
+	timeout connect 1s
+	timeout client  1s
+	timeout server  1s
+
+frontend fe
+	bind "fd@${fe}"
+
+	 requests
+	http-request set-var(txn.iif) req.hdr_cnt(count),iif(ok,ko)
+	http-response set-header iif %[var(txn.iif)]
+
+	default_backend be
+
+backend be
+	server s1 ${s1_addr}:${s1_port}
+} -start
+
+client c1 -connect ${h1_fe_sock} {
+	txreq
+	rxresp
+	expect resp.status == 200
+	expect resp.http.iif == "ko"
+	txreq \
+		-hdr "count: 1"
+	rxresp
+	expect resp.status == 200
+	expect resp.http.iif == "ok"
+	txreq \
+		-hdr "count: 1,2"
+	rxresp
+	expect resp.status == 200
+	expect resp.http.iif == "ok"
+} -run
diff --git a/src/sample.c b/src/sample.c
index 3a1534865..a9c08ef54 100644
--- a/src/sample.c
+++ b/src/sample.c
@@ -3121,6 +3121,26 @@ static int sample_conv_secure_memcmp(const struct arg *arg_p, struct sample *smp
 }
 #endif
 
+/* Takes a boolean as input. Returns the first argument if that boolean is true and
+ * the second argument otherwise.
+ */
+static int sample_conv_iif(const struct arg *arg_p, struct sample *smp, void *private)
+{
+	smp->data.type = SMP_T_STR;
+	smp->flags |= SMP_F_CONST;
+
+	if (smp->data.u.sint) {
+		smp->data.u.str.data = arg_p[0].data.str.data;
+		smp->data.u.str.area = arg_p[0].data.str.area;
+	}
+	else {
+		smp->data.u.str.data = arg_p[1].data.str.data;
+		smp->data.u.str.area = arg_p[1].data.str.area;
+	}
+
+	return 1;
+}
+
 #define GRPC_MSG_COMPRESS_FLAG_SZ 1 /* 1 byte */
 #define GRPC_MSG_LENGTH_SZ4 /* 4 bytes */
 #define GRPC_MSG_HEADER_SZ(GRPC_MSG_COMPRESS_FLAG_SZ + GRPC_MSG_LENGTH_SZ)
@@ -3782,6 +3802,8 

Re: [PATCH] MINOR: Add either(,) converter

2020-09-11 Thread Willy Tarreau
Hi guys,

First, I really like the feature, that's a great idea.

On Fri, Sep 11, 2020 at 04:28:31PM +0200, Miroslav Zagorac wrote:
> On 09/11/2020 03:56 PM, Tim Düsterhus, WoltLab GmbH wrote:
> > We've had a bit of discussion regarding the naming of the converter. I
> > wanted to avoid calling it `if`, because then we could have stuff like this:
> > 
> >http-request set-var(txn.foo) bool(1),if(bar,baz)
> > 
> > which can easily be confused with:
> > 
> >http-request set-var(txn.foo) bool(1) if bar
> > 
> > Some other naming suggestions that came up:
> > 
> > - choice (my initial choice)
> > - ifor / if_or
> > - ifelse / if_else
> > - iftrue (with the  argument being optional)
> 
> Maybe something like this would be appropriate (IIF)?

I was thinking about "select" which I've seldom seen, I never heard about
"iif" previously but it has the benefit of being quite short (convenient
for long expressions) and already in use elsewhere. And it also has the
benefit that someone searching for "if" would easily stumble on it.

I'm fine with the patch BTW, so if everyone agrees on "iif", let's suggest
we go with it ?

thanks,
Willy



Re: [PATCH] MINOR: Add either(,) converter

2020-09-11 Thread Miroslav Zagorac

On 09/11/2020 03:56 PM, Tim Düsterhus, WoltLab GmbH wrote:

We've had a bit of discussion regarding the naming of the converter. I
wanted to avoid calling it `if`, because then we could have stuff like this:

   http-request set-var(txn.foo) bool(1),if(bar,baz)

which can easily be confused with:

   http-request set-var(txn.foo) bool(1) if bar

Some other naming suggestions that came up:

- choice (my initial choice)
- ifor / if_or
- ifelse / if_else
- iftrue (with the  argument being optional)



Maybe something like this would be appropriate (IIF)?

https://en.wikipedia.org/wiki/IIf

--
Zaga

What can change the nature of a man?



[PATCH] MINOR: Add either(,) converter

2020-09-11 Thread Tim Düsterhus , WoltLab GmbH
Willy,

[keep this email in CC, it's not subscribed to the list]

"either() takes a boolean as input and returns one of the two argument
strings depending on whether the boolean is true."

Find the full details in the attached patch.

---

We've had a bit of discussion regarding the naming of the converter. I
wanted to avoid calling it `if`, because then we could have stuff like this:

  http-request set-var(txn.foo) bool(1),if(bar,baz)

which can easily be confused with:

  http-request set-var(txn.foo) bool(1) if bar

Some other naming suggestions that came up:

- choice (my initial choice)
- ifor / if_or
- ifelse / if_else
- iftrue (with the  argument being optional)

I'm happy to adjust the patch based on your preferences.

I'd also appreciate if this one could make a backport into 2.2. But I
would fully understand if you refuse to backport this into a LTS branch.
November is not too far away :-)

Best regards
Tim Düsterhus
Developer WoltLab GmbH

-- 

WoltLab GmbH
Nedlitzer Str. 27B
14469 Potsdam

Tel.: +49 331 96784338

duester...@woltlab.com
www.woltlab.com

Managing director:
Marcel Werk

AG Potsdam HRB 26795 P
>From 52e50548f97a795cb891adf0f18a8ebc1e38b188 Mon Sep 17 00:00:00 2001
From: Tim Duesterhus 
Date: Fri, 11 Sep 2020 14:25:23 +0200
Subject: [PATCH] MINOR: Add either(,) converter
To: haproxy@formilux.org
Cc: w...@1wt.eu

either() takes a boolean as input and returns one of the two argument
strings depending on whether the boolean is true.

This converter most likely is most useful to return the proper scheme
depending on the value returned by the `ssl_fc` fetch, e.g. for use within
the `x-forwarded-proto` request header.

However it can also be useful for use within a template that is sent to
the client using `http-request return` with a `lf-file`. It allows the
administrator to implement a simple condition, without needing to prefill
variables within the regular configuration using `http-request
set-var(req.foo)`.
---
 doc/configuration.txt  |  7 ++
 reg-tests/converter/either.vtc | 45 ++
 src/sample.c   | 22 +
 3 files changed, 74 insertions(+)
 create mode 100644 reg-tests/converter/either.vtc

diff --git a/doc/configuration.txt b/doc/configuration.txt
index c1f6f8219..197b5cec5 100644
--- a/doc/configuration.txt
+++ b/doc/configuration.txt
@@ -15094,6 +15094,13 @@ djb2([])
   32-bit hash is trivial to break. See also "crc32", "sdbm", "wt6", "crc32c",
   and the "hash-type" directive.
 
+either(,)
+  Returns the  string if the input value is true. Returns the 
+  string otherwise.
+
+  Example:
+http-request set-header x-forwarded-proto %[ssl_fc,either(https,http)]
+
 even
   Returns a boolean TRUE if the input value of type signed integer is even
   otherwise returns FALSE. It is functionally equivalent to "not,and(1),bool".
diff --git a/reg-tests/converter/either.vtc b/reg-tests/converter/either.vtc
new file mode 100644
index 0..8a155c356
--- /dev/null
+++ b/reg-tests/converter/either.vtc
@@ -0,0 +1,45 @@
+varnishtest "either converter Test"
+
+feature ignore_unknown_macro
+
+server s1 {
+	rxreq
+	txresp
+} -repeat 3 -start
+
+haproxy h1 -conf {
+defaults
+	mode http
+	timeout connect 1s
+	timeout client  1s
+	timeout server  1s
+
+frontend fe
+	bind "fd@${fe}"
+
+	 requests
+	http-request set-var(txn.either) req.hdr_cnt(count),either(ok,ko)
+	http-response set-header either %[var(txn.either)]
+
+	default_backend be
+
+backend be
+	server s1 ${s1_addr}:${s1_port}
+} -start
+
+client c1 -connect ${h1_fe_sock} {
+	txreq
+	rxresp
+	expect resp.status == 200
+	expect resp.http.either == "ko"
+	txreq \
+		-hdr "count: 1"
+	rxresp
+	expect resp.status == 200
+	expect resp.http.either == "ok"
+	txreq \
+		-hdr "count: 1,2"
+	rxresp
+	expect resp.status == 200
+	expect resp.http.either == "ok"
+} -run
diff --git a/src/sample.c b/src/sample.c
index 3a1534865..d6baf6c89 100644
--- a/src/sample.c
+++ b/src/sample.c
@@ -3121,6 +3121,26 @@ static int sample_conv_secure_memcmp(const struct arg *arg_p, struct sample *smp
 }
 #endif
 
+/* Takes a boolean as input. Returns the first argument if that boolean is true and
+ * the second argument otherwise.
+ */
+static int sample_conv_either(const struct arg *arg_p, struct sample *smp, void *private)
+{
+	smp->data.type = SMP_T_STR;
+	smp->flags |= SMP_F_CONST;
+
+	if (smp->data.u.sint) {
+		smp->data.u.str.data = arg_p[0].data.str.data;
+		smp->data.u.str.area = arg_p[0].data.str.area;
+	}
+	else {
+		smp->data.u.str.data = arg_p[1].data.str.data;
+		smp->data.u.str.area = arg_p[1].data.str.area;
+	}
+
+	return 1;
+}
+
 #define GRPC_MSG_COMPRESS_FLAG_SZ 1 /* 1 byte */
 #define GRPC_MSG_LENGTH_SZ4 /* 4 bytes */
 #define GRPC_MSG_HEADER_SZ(GRPC_MSG_COMPRESS_FLAG_SZ + GRPC_MSG_LENGTH_SZ)
@@ -3782,6 +3802,8 @@ static struct sample_conv_kw_list sample_conv_kws = {ILH, {
 	{ "ungrpc", sample_conv_ungrpc,ARG2(1,PBUF_FNUM,STR), 

Re: http2 smuggling

2020-09-11 Thread Willy Tarreau
On Fri, Sep 11, 2020 at 09:56:21AM +0200, Tim Düsterhus wrote:
> Willy,
> 
> Am 11.09.20 um 09:42 schrieb Willy Tarreau:
> > On Fri, Sep 11, 2020 at 09:02:57AM +0200, Tim Düsterhus wrote:
> >> According to the article performing a h2c upgrade via TLS is not valid
> >> according to the spec. HAProxy implements the H2 spec.
> > 
> > "according to the article" :-) There's no such mention in the spec
> > itself from what I remember, it's just that it's usually pointless,
> > but there may be a lot of situations where it's considered better to
> > forward an upgradable connection over TLS to the next intermediary
> > because the intermediary network is not safe.
> 
> I might misunderstand it, but I'd say that RFC 7540#3.3 specifically
> disallows h2c for TLS:
> 
> >HTTP/2 over TLS uses the "h2" protocol identifier.  The "h2c"
> >protocol identifier MUST NOT be sent by a client or selected by a
> >server; the "h2c" protocol identifier describes a protocol that does
> >not use TLS.

Ah OK, got it, this one is about the protocol indicated in ALPN, as
reflected by the preceeding sentence: "A client that makes a request
to an "https" URI uses TLS with the application-layer protocol
negotiation (ALPN) extension".

When this was designed, there were long discussions about whether h2
alone or both h2/h2c had to be registered as protocols, and about
whether h2c was allowed in ALPN and in this case what to do about it.
It was then concluded that the easiest solution was to state that h2c
is the cleartext upgrade version while h2 is the raw one over TLS.

> >> Question 1: Should HAProxy reject requests that set Upgrade: h2c over
> >> TLS? I think it should. Basically the following rule should be applied
> >> automatically to my understanding.
> >>
> >> http-request deny deny_status 400 if { req.hdr(upgrade) h2c } { ssl_fc }
> > 
> > No I disagree. Let's say you have an h2c client on datacenter 1 and
> > an h2c server on datacenter 2. This rule would prevent you from using
> > the local haproxy to secure the connection, while providing zero benefit.
> 
> If I just want to secure the connection I would use 'mode tcp' where
> HAProxy is a dumb pipe and this rule would not apply.

You can't always. Let's say haproxy is your local side car and your
application sends everything to it. You'll route requests to various
places according to their Host and URI. Why is it that for a particular
reason one type of request is considered special enough to deserve being
broken ?

> > By the way, it's fun to see that a discussion started a few days ago
> > regarding the uselessness of h2c and its removal from the next H2 spec
> > because "nobody implemented it yet" :-)  And actually the guy had to
> > implement its own server to find a complying one.
> 
> I believe nginx can do h2c.

Maybe but then probably not by default, given that the guy used it in
his test to forward the Upgrade request. If it would support it by
default, it would have considered that request for itself instead. I'm
pretty sure that nginx does support cleartext h2 (without TLS) however.

By the way if you're interested in the todo list that starts to be
collected for an update of the H2 spec, it's here at the moment:

   https://github.com/martinthomson/http2-spec/issues/

Cheers,
Willy



Re: http2 smuggling

2020-09-11 Thread Tim Düsterhus
Willy,

Am 11.09.20 um 09:42 schrieb Willy Tarreau:
> On Fri, Sep 11, 2020 at 09:02:57AM +0200, Tim Düsterhus wrote:
>> According to the article performing a h2c upgrade via TLS is not valid
>> according to the spec. HAProxy implements the H2 spec.
> 
> "according to the article" :-) There's no such mention in the spec
> itself from what I remember, it's just that it's usually pointless,
> but there may be a lot of situations where it's considered better to
> forward an upgradable connection over TLS to the next intermediary
> because the intermediary network is not safe.

I might misunderstand it, but I'd say that RFC 7540#3.3 specifically
disallows h2c for TLS:

>HTTP/2 over TLS uses the "h2" protocol identifier.  The "h2c"
>protocol identifier MUST NOT be sent by a client or selected by a
>server; the "h2c" protocol identifier describes a protocol that does
>not use TLS.

-

>> Question 1: Should HAProxy reject requests that set Upgrade: h2c over
>> TLS? I think it should. Basically the following rule should be applied
>> automatically to my understanding.
>>
>> http-request deny deny_status 400 if { req.hdr(upgrade) h2c } { ssl_fc }
> 
> No I disagree. Let's say you have an h2c client on datacenter 1 and
> an h2c server on datacenter 2. This rule would prevent you from using
> the local haproxy to secure the connection, while providing zero benefit.

If I just want to secure the connection I would use 'mode tcp' where
HAProxy is a dumb pipe and this rule would not apply.

> By the way, it's fun to see that a discussion started a few days ago
> regarding the uselessness of h2c and its removal from the next H2 spec
> because "nobody implemented it yet" :-)  And actually the guy had to
> implement its own server to find a complying one.

I believe nginx can do h2c.

Best regards
Tim Düsterhus



Re: http2 smuggling

2020-09-11 Thread Willy Tarreau
On Fri, Sep 11, 2020 at 09:02:57AM +0200, Tim Düsterhus wrote:
> According to the article performing a h2c upgrade via TLS is not valid
> according to the spec. HAProxy implements the H2 spec.

"according to the article" :-) There's no such mention in the spec
itself from what I remember, it's just that it's usually pointless,
but there may be a lot of situations where it's considered better to
forward an upgradable connection over TLS to the next intermediary
because the intermediary network is not safe.

> Question 1: Should HAProxy reject requests that set Upgrade: h2c over
> TLS? I think it should. Basically the following rule should be applied
> automatically to my understanding.
> 
> http-request deny deny_status 400 if { req.hdr(upgrade) h2c } { ssl_fc }

No I disagree. Let's say you have an h2c client on datacenter 1 and
an h2c server on datacenter 2. This rule would prevent you from using
the local haproxy to secure the connection, while providing zero benefit.

By the way, it's fun to see that a discussion started a few days ago
regarding the uselessness of h2c and its removal from the next H2 spec
because "nobody implemented it yet" :-)  And actually the guy had to
implement its own server to find a complying one.

> Further the article says that the HTTP2-Settings header is a hop by hop
> header. It should not be forwarded by a proxy. According to the article
> HAProxy *does* forward it.
> 
> Question 2: Should HAProxy automatically strip the HTTP2-Settings header
> when forwarding requests?

Haproxy is not a proxy but a gateway. See it as a transparent cable
capable of staying synchronized with both ends and interacting the least
possible. There's nothing wrong with repeating hop-by-hop header fields
on gateways if you don't break the transport nor semantics, it's exactly
what happens when you use a raw TLS offloader for example, which doesn't
even understand HTTP. Technically speaking we're not "keeping" the
header, we're formulating a new request that places it again since it's
compatible with both sides' capabilities. In fact haproxy produces a
new connection header with all the tokens that it did not use, since
they are by definition for the next hop. "keep-alive" and "close" are
the only two that are of interest to us and that are terminated locally.

For example, the TE header is hop-by-hop. Haproxy doesn't need it but
maintains the transport, messaging and semantics end-to-end so it does
not need to eliminate it and disrupt end-to-end connectivity as long as
it stays synchronized. Really that's not different from a pure TCP proxy
(which is how haproxy started 20 years ago by the way).

Hoping this clarifies the point,
Willy



Re: http2 smuggling

2020-09-11 Thread Willy Tarreau
On Fri, Sep 11, 2020 at 02:52:30AM -0400, John Lauro wrote:
> I could be wrong, but I think he is stating that if you have that
> allowed, it can be used to get a direct connection to the backend
> bypassing any routing or acls you have in the load balancer, so if you
> some endpoints are blocked, or internal only, they could potentially
> be accessed this way.
> For example, if you have something like:
>   acl is_restrict path_sub /.git/
>   http-request deny if is_restrict !is_safe_ip
> 
> The acl could be bypassed by using the method to connect directly to a 
> backend.
> 
> That's not to say it's a security flaw in haproxy, but a potential
> misconfiguration that could allow traffic you thought was blocked by
> the proxy.

We're talking about an upgrade agreement between the client and the
server in order to use a protocol that the LB doesn't speak. This is
typically used for websocket, I remember having seen one terminal
server using this as well, maybe RDP or Citrix, I don't remember.

This is exactly the same as CONNECT+200: both ends agree to upgrade the
HTTP connection to another protocol till it ends. It's not HTTP that
applies once the tunnel is set up, so there are no filtering rules
nor whatever, exactly like after an accepted CONNECT request.

Willy



Re: [*EXT*] Re: http2 smuggling

2020-09-11 Thread Willy Tarreau
Hi Ionel,

On Fri, Sep 11, 2020 at 08:35:58AM +0200, Ionel GARDAIS wrote:
> Hi Willy,
> 
> Being devil's advocate : isn't the point that even if this is a documented,
> standardized and intended behavior, users relying on the reverse proxy for
> security/sanity checks could by tricked by this feature inadvertently ?

The security checks are properly performed during the request for upgrade,
the reporter even mentioned it in his article where he blocks the upgrade
by removing the Upgrade header. Note that the important point is that the
server *must* accept the upgrade for this to work. So there's no real
accident here. It's exactly what's being used for websocket.

Willy



Re: http2 smuggling

2020-09-11 Thread Tim Düsterhus
Willy,

Am 11.09.20 um 08:07 schrieb Willy Tarreau:
> On Fri, Sep 11, 2020 at 01:55:10PM +1000, Igor Cicimov wrote:
>> Should we be worried?
>>
>> https://portswigger.net/daily-swig/http-request-smuggling-http-2-opens-a-new-attack-tunnel
> 
> But this stuff is total non-sense. Basically the guy is complaining
> that the products he tested work exactly as desired, designed and
> documented!
> 
> The principle of the upgrade at the gateway level precisely is to say
> "OK both the client and the server want to speak another protocol you
> agreed upon, let me retract" and let them talk over a tunnel. That's
> exactly what is needed to support WebSocket for example. The simple
> fact that he found that many proxies/gateways work like this should
> ring a bell about the intended behavior!
> 

I'm very well aware about your opinion regarding access control at the
edge by now, however I have 2 questions at this point.

I've read the official write-up of the findings of the researcher:
https://labs.bishopfox.com/tech-blog/h2c-smuggling-request-smuggling-via-http/2-cleartext-h2c.
It contains quite a few more details and I recommend to take a look at
it to answer my questions.

According to the article performing a h2c upgrade via TLS is not valid
according to the spec. HAProxy implements the H2 spec.

Question 1: Should HAProxy reject requests that set Upgrade: h2c over
TLS? I think it should. Basically the following rule should be applied
automatically to my understanding.

http-request deny deny_status 400 if { req.hdr(upgrade) h2c } { ssl_fc }



Further the article says that the HTTP2-Settings header is a hop by hop
header. It should not be forwarded by a proxy. According to the article
HAProxy *does* forward it.

Question 2: Should HAProxy automatically strip the HTTP2-Settings header
when forwarding requests?

Best regards
Tim Düsterhus



Re: http2 smuggling

2020-09-11 Thread John Lauro
I could be wrong, but I think he is stating that if you have that
allowed, it can be used to get a direct connection to the backend
bypassing any routing or acls you have in the load balancer, so if you
some endpoints are blocked, or internal only, they could potentially
be accessed this way.
For example, if you have something like:
  acl is_restrict path_sub /.git/
  http-request deny if is_restrict !is_safe_ip

The acl could be bypassed by using the method to connect directly to a backend.

That's not to say it's a security flaw in haproxy, but a potential
misconfiguration that could allow traffic you thought was blocked by
the proxy.


On Fri, Sep 11, 2020 at 2:07 AM Willy Tarreau  wrote:
>
> Hi Igor,
>
> On Fri, Sep 11, 2020 at 01:55:10PM +1000, Igor Cicimov wrote:
> > Should we be worried?
> >
> > https://portswigger.net/daily-swig/http-request-smuggling-http-2-opens-a-new-attack-tunnel
>
> But this stuff is total non-sense. Basically the guy is complaining
> that the products he tested work exactly as desired, designed and
> documented!
>
> The principle of the upgrade at the gateway level precisely is to say
> "OK both the client and the server want to speak another protocol you
> agreed upon, let me retract" and let them talk over a tunnel. That's
> exactly what is needed to support WebSocket for example. The simple
> fact that he found that many proxies/gateways work like this should
> ring a bell about the intended behavior!
>
> In addition there is zero smuggling here as there is no desynchronisation.
> It's just a tunnel between the client and the server, both agreeing to
> do so. It does *exactly* the same as if the client had connected to the
> server using a CONNECT method and the server had returned 200. So there
> is absolutely no attack nor whatever here, just a connection that remains
> dedicated to a client and a server till the end.
>
> Sadly, as usual after people discover protocols during the summer, some
> journalists will surely want to make noise about this to put some bread
> on their table...
>
> Thanks for the link anyway I had a partial laugh; partial only because
> it makes useless noise.
>
> Cheers,
> Willy
>



Re: [*EXT*] Re: http2 smuggling

2020-09-11 Thread Ionel GARDAIS
Hi Willy,

Being devil's advocate : isn't the point that even if this is a documented, 
standardized and intended behavior, users relying on the reverse proxy for 
security/sanity checks could by tricked by this feature inadvertently ?

-- 
Ionel GARDAIS
Tech'Advantage CIO - IT Team manager

- Mail original -
De: "Willy Tarreau" 
À: "Igor Cicimov" 
Cc: "haproxy" 
Envoyé: Vendredi 11 Septembre 2020 08:19:12
Objet: [*EXT*] Re: http2 smuggling

On Fri, Sep 11, 2020 at 08:07:02AM +0200, Willy Tarreau wrote:
> Sadly, as usual after people discover protocols during the summer, some
> journalists will surely want to make noise about this to put some bread
> on their table...
> 
> Thanks for the link anyway I had a partial laugh; partial only because
> it makes useless noise.

And sadly, this one already started to make some noise there about his
recent discovery of a 20-years old standard:

   https://twitter.com/theBumbleSec

Had he asked if we supported 101, we could even have saved him time
in his HTTP discover test by pointing him to the doc:

   
http://git.haproxy.org/?p=haproxy.git;a=blob;f=doc/configuration.txt;h=c1f6f82;hb=HEAD#l332

Probably that next year he will discover that we also support CONNECT.
It's not even funny, the world is really doomed...

Willy
--
232 avenue Napoleon BONAPARTE 92500 RUEIL MALMAISON
Capital EUR 219 300,00 - RCS Nanterre B 408 832 301 - TVA FR 09 408 832 301




Re: http2 smuggling

2020-09-11 Thread Willy Tarreau
On Fri, Sep 11, 2020 at 08:07:02AM +0200, Willy Tarreau wrote:
> Sadly, as usual after people discover protocols during the summer, some
> journalists will surely want to make noise about this to put some bread
> on their table...
> 
> Thanks for the link anyway I had a partial laugh; partial only because
> it makes useless noise.

And sadly, this one already started to make some noise there about his
recent discovery of a 20-years old standard:

   https://twitter.com/theBumbleSec

Had he asked if we supported 101, we could even have saved him time
in his HTTP discover test by pointing him to the doc:

   
http://git.haproxy.org/?p=haproxy.git;a=blob;f=doc/configuration.txt;h=c1f6f82;hb=HEAD#l332

Probably that next year he will discover that we also support CONNECT.
It's not even funny, the world is really doomed...

Willy



Re: http2 smuggling

2020-09-11 Thread Willy Tarreau
Hi Igor,

On Fri, Sep 11, 2020 at 01:55:10PM +1000, Igor Cicimov wrote:
> Should we be worried?
> 
> https://portswigger.net/daily-swig/http-request-smuggling-http-2-opens-a-new-attack-tunnel

But this stuff is total non-sense. Basically the guy is complaining
that the products he tested work exactly as desired, designed and
documented!

The principle of the upgrade at the gateway level precisely is to say
"OK both the client and the server want to speak another protocol you
agreed upon, let me retract" and let them talk over a tunnel. That's
exactly what is needed to support WebSocket for example. The simple
fact that he found that many proxies/gateways work like this should
ring a bell about the intended behavior!

In addition there is zero smuggling here as there is no desynchronisation.
It's just a tunnel between the client and the server, both agreeing to
do so. It does *exactly* the same as if the client had connected to the
server using a CONNECT method and the server had returned 200. So there
is absolutely no attack nor whatever here, just a connection that remains
dedicated to a client and a server till the end.

Sadly, as usual after people discover protocols during the summer, some
journalists will surely want to make noise about this to put some bread
on their table...

Thanks for the link anyway I had a partial laugh; partial only because
it makes useless noise.

Cheers,
Willy