[PR] BUG/MINOR: http-fetch: recognize IPv6 addresses in square brackets in req.hdr_ip()
Dear list! Author: Oto Valek Number of patches: 2 This is an automated relay of the Github pull request: BUG/MINOR: http-fetch: recognize IPv6 addresses in square brackets in req.hdr_ip() Patch title(s): BUG/MINOR: http-fetch: recognize IPv6 addresses in square brackets in req.hdr_ip() REGTEST: enclose one of the IPv6 address in square brackets Link: https://github.com/haproxy/haproxy/pull/2055 Edit locally: wget https://github.com/haproxy/haproxy/pull/2055.patch && vi 2055.patch Apply locally: curl https://github.com/haproxy/haproxy/pull/2055.patch | git am - Description: Fixes haproxy/haproxy#2054 To comply with [RFC7239](https://www.rfc-editor.org/rfc/rfc7239.html#section-6.1) and [RFC3986](https://www.rfc-editor.org/rfc/rfc3986.html#section-3.2.2), the `req.hdr_ip()` should recognize IPv6 addresses in square brackets. As the `inet_pton()` call does not support this format, the `smp_fetch_hdr_ip()` function was changed to trim possible `'['` and `']'` before calling `inet_pton()`. Reg tests were updated to cover this case. Instructions: This github pull request will be closed automatically; patch should be reviewed on the haproxy mailing list (haproxy@formilux.org). Everyone is invited to comment, even the patch's author. Please keep the author and list CCed in replies. Please note that in absence of any response this pull request will be lost.
Re: Can you block this?
On Fri, Feb 24, 2023 at 10:18:14AM -0700, Bryan Arenal wrote:
> And would this work to reject any request that has the
> 'X-Forwarded-For' header?
>
> acl is-forwarded hdr_sub(x-forwarded-for)
> http-request reject if is-forwarded
No, not like this, as you're searching for sub-strings in this header
name but provide no substring. Instead just use the "found" match
method, which only searches for the element:
acl is-forwarded hdr(x-forwarded-for) -m found
http-request reject if is-forwarded
or:
http-request reject if { hdr(x-forwarded-for) -m found }
Note that I tend to find it more convenient to use anonymous ACLs like
above for simple definitions, especially when you're dealing with attacks
that tend to quickly require a lot of patterns, and when ACL names can
quickly become misleading. However declaring ACLs by names remains much
more convenient when you're starting to combine them. I think that
"is-forwarded" is sufficiently self-explanatory and definitely satisfies
this use case, but it was just to give an example.
> How resource intensive do you think this would this be?
It's very light. Just to give you an example, on my laptop, with one
single core assigned to haproxy, a config matching this rule achieves
133000 connections per second on a single core. And when the rule does
not match, just searching for it lowers the perf from 198k to 192k RPS
per core, or 158 nanoseconds of CPU for the whole rule evaluation, a
part of which is amortized if several rules are evaluated. 10 of them
only consume 832 ns here so the cost of starting evaluation is 75ns then
83ns per rule. It's reasonable to use a handful of such rules to fight
a DDoS if you need. The most important is to provide the least possible
information to the attacker about what you're detecting and how you
proceed (e.g. silent-drop and tarpit are great for this).
Willy
Re: Can you block this?
On Fri, Feb 24, 2023 at 12:33 AM Willy Tarreau wrote: > > On Fri, Feb 24, 2023 at 05:39:13AM +, Robin H. Johnson wrote: > > On Thu, Feb 23, 2023 at 06:48:14PM -0700, Bryan Arenal wrote: > > > Hi there, > > > > > > I'm seeing some traffic from what appears to be bad actors and am > > > wanting to block them. I see this in the existing config but being > > > new to haproxy, it doesn't seem like it's configured correctly but I'm > > > not sure: > > > > > > frontend main > > > bind :80 > > > acl bad_ip src > > > acl bad_ip_proxy hdr_ip(X-Forwarded-For) > > Off the top of my head, this should probably be: > > acl bad_ip src -f /etc/haproxy/blocklist.lst > > acl bad_ip_proxy hdr_ip(X-Forwarded-For) -f /etc/haproxy/blocklist.lst > > > > > tcp-request connection reject if bad_ip || bad_ip_proxy > > I'm not sure offhand about the processing order for the header case. > > > > You might need BOTH: > > tcp-request connection reject if bad_ip || bad_ip_proxy > > http-request connection reject if bad_ip || bad_ip_proxy > > In fact the tcp-request rule will not consider the bad_ip_proxy ACL since > it relies on header extraction that is not possible at this step. Indeed, > You'd need to deal with this using the http-request rule. I would also > suggest that if the IP is blocked at the TCP level then it's not needed > to evaluate it again at the HTTP layer. This would give: > > tcp-request connection reject if bad_ip > http-request reject if bad_ip_proxy > > > Depending on the scale of the traffic, the one problem you'll have here is > > that HAProxy still has to process the problematic requests. > > In fact it depends. For example if it's a POST with a heavy body, a "deny" > rule will return a 403 and will drain the rest of the body in order to try > to recover the connection, but a "reject" rule will instantly break the > connection, which instantly stops any processing. > > > In that case I suggest writing a feedback loop that adds the bad IP to an > > ipset > > set, to block the traffic before it gets to HAProxy, for some period of > > time. > > The trigger for the loop can either be a tail on the logfile, or using some > > variation of the set* functionality (set-acl, set-map, set-mark) and > > exporting > > the data to ipset. > > I used to proceed like this a very long time ago, and found that it was > not always convenient due to the lack of observability (i.e. "is that rule > still needed or has the attack stopped?"). Given the performance of most > machines nowadays, I *tend* to prefer only blocking using the ACLs in TCP > rules (that's still very fast). However you're right, sometimes it can be > required to block IPs at the packet level, particularly for multi-protocol > attacks that span over many ports. > > By the way, if attacks come from a small to medium sized set of clients, > the "tarpit" and "silent-drop" actions can be more effective because they > consume resources on the client by keeping the connection open. But *do not* > do that if there is a firewall or NAT box between the net and your LB or > it might not find this funny. For silent-drop there's a new "rst-ttl" > option that allows to send a reset with a limited TTL that only the > firewall/NAT box will see. This one might be considered as an option. Thank you both VERY much for your help -- I really appreciate it. That makes total sense that you'd have to use http-request rather than tcp-request since you've got to evaluate the headers. And would this work to reject any request that has the 'X-Forwarded-For' header? acl is-forwarded hdr_sub(x-forwarded-for) http-request reject if is-forwarded How resource intensive do you think this would this be? Thanks again!
Re: Host header not copied to :authority when using a proto h2 backend
Hello Willy. Thank you very much for the detailed answer. I have reported this in Jetty issue tracker, I hope they can support this use case: https://github.com/eclipse/jetty.project/issues/9436 Kind regards, Óscar On Fri, Feb 24, 2023 at 2:44 PM Willy Tarreau wrote: > Hello, > > On Fri, Feb 24, 2023 at 02:19:30PM +0100, Óscar Frías Barranco wrote: > > Hello > > > > I am using haproxy 2.4.18 with a frontend configured with alpn > h2,http/1.1 > > > > The problem that I am facing is that if I add "proto h2" to the backends, > > when a remote client connects to the frontend using HTTP 1.1, the request > > is sent to the backend server without copying the "Host" header to the > > ":authority" pseudo-header. > > This is normal and expected, it complies with the spec that is pretty > strict about this: > >https://www.rfc-editor.org/rfc/rfc7540#section-8.1.2.3 > > The ":authority" pseudo-header field includes the authority > portion of the target URI ([RFC3986], Section 3.2). The authority > MUST NOT include the deprecated "userinfo" subcomponent for "http" > or "https" schemed URIs. > > To ensure that the HTTP/1.1 request line can be reproduced > accurately, this pseudo-header field MUST be omitted when > translating from an HTTP/1.1 request that has a request target in > origin or asterisk form (see [RFC7230], Section 5.3). > > and was further refined in the updated spec: > >https://www.rfc-editor.org/rfc/rfc9113#name-request-pseudo-header-field > >The ":authority" pseudo-header field conveys the authority portion > (Section >3.2 of [RFC3986]) of the target URI (Section 7.1 of [HTTP]). >(...) >An intermediary that forwards a request over HTTP/2 MUST construct an >":authority" pseudo-header field using the authority information from > the >control data of the original request, unless the original request's > target URI >does not contain authority information (in which case it MUST NOT > generate >":authority"). Note that the Host header field is not the sole source > of this >information; see Section 7.2 of [HTTP]. > > If the HTTP/1 request contains an authority part, it will appear there. > Otherwise only the host is used. In summary, the H2 request looks very > close to the H1 one: Host conveys Host, authority conveys authority. > > > This causes the application server (Jetty in my case) to report the > backend > > server numeric IP address as the server name instead of reporting the web > > domain (which is in the Host header) as one would expect. The URI > reported > > at the application level is also wrong (numeric instead of the domain) > for > > the same reason I suppose. > > Given that :authority is not mandatory, I presume there should be at > least an option in Jetty to report the Host when it is missing. For > example RFC7540 used to hint about this: > > Clients > that generate HTTP/2 requests directly SHOULD use the ":authority" > pseudo-header field instead of the Host header field. An > intermediary that converts an HTTP/2 request to HTTP/1.1 MUST > create a Host header field if one is not present in a request by > copying the value of the ":authority" pseudo-header field. > > > If the remote user connects to the frontend with HTTP/2 everything is > > correct (I assume that is because the :authority pseudo header was > already > > in there). > > > > If I remove "proto h2" from the backend, everything is OK too (because in > > this case :authority is converted to a Host header when connecting to the > > backend). > > > > Is there something I can do to fix this? Is this a bug (either from > > haproxy or Jetty) ? > > I suspect it's not a bug but a config decision on Jetty's side. > > Particularly, a few years ago some vulnerabilities were exposed in H2 > agents not strictly following the spec regarding host vs :authority, > making combinations of such agents rather "fun". It's very possible > that some implementations have decided to restrict the fallbacks by > default and to only rely on :authority unless configured to look at > Host when missing, according to the spec. > > If required there is a hack that can be used to force an authority. > If you force an authority in your request it should normally work: > >http-request set-uri https://%[req.hdr(host)]%[pathq] > > But it would be cleaner to make sure the server accepts a request that > only has a host and no authority. > > Regards, > Willy >
RE: Marketing list RSA 2023
Hi, I hope my emails are reaching your inbox! If are you still interested to acquire the attendee contacts! I can get back to you with pricing and counts. Regards, Teresa Turner Event coordinator From: Teresa Turner Sent: Thursday, February 16, 2023 10:55 AM To: haproxy@formilux.org Subject: Marketing list RSA 2023 Have a great day! We are offering you the pre-registered attendee contacts for RSA Conference 2023. List includes: Contact Name, Title, Company Name, Website, Physical Address, Phone Number, Fax Number, SIC, Industry type, Employees, Direct Emails, and Verification results. If you would like to acquire the list, I am happy to share cost and counts. Awaiting your reply. Cheers, Teresa Turner "Event coordinator- RSA 2023" If you are not interested to receive future emails reply as "take-out"
Re: Host header not copied to :authority when using a proto h2 backend
Hello, On Fri, Feb 24, 2023 at 02:19:30PM +0100, Óscar Frías Barranco wrote: > Hello > > I am using haproxy 2.4.18 with a frontend configured with alpn h2,http/1.1 > > The problem that I am facing is that if I add "proto h2" to the backends, > when a remote client connects to the frontend using HTTP 1.1, the request > is sent to the backend server without copying the "Host" header to the > ":authority" pseudo-header. This is normal and expected, it complies with the spec that is pretty strict about this: https://www.rfc-editor.org/rfc/rfc7540#section-8.1.2.3 The ":authority" pseudo-header field includes the authority portion of the target URI ([RFC3986], Section 3.2). The authority MUST NOT include the deprecated "userinfo" subcomponent for "http" or "https" schemed URIs. To ensure that the HTTP/1.1 request line can be reproduced accurately, this pseudo-header field MUST be omitted when translating from an HTTP/1.1 request that has a request target in origin or asterisk form (see [RFC7230], Section 5.3). and was further refined in the updated spec: https://www.rfc-editor.org/rfc/rfc9113#name-request-pseudo-header-field The ":authority" pseudo-header field conveys the authority portion (Section 3.2 of [RFC3986]) of the target URI (Section 7.1 of [HTTP]). (...) An intermediary that forwards a request over HTTP/2 MUST construct an ":authority" pseudo-header field using the authority information from the control data of the original request, unless the original request's target URI does not contain authority information (in which case it MUST NOT generate ":authority"). Note that the Host header field is not the sole source of this information; see Section 7.2 of [HTTP]. If the HTTP/1 request contains an authority part, it will appear there. Otherwise only the host is used. In summary, the H2 request looks very close to the H1 one: Host conveys Host, authority conveys authority. > This causes the application server (Jetty in my case) to report the backend > server numeric IP address as the server name instead of reporting the web > domain (which is in the Host header) as one would expect. The URI reported > at the application level is also wrong (numeric instead of the domain) for > the same reason I suppose. Given that :authority is not mandatory, I presume there should be at least an option in Jetty to report the Host when it is missing. For example RFC7540 used to hint about this: Clients that generate HTTP/2 requests directly SHOULD use the ":authority" pseudo-header field instead of the Host header field. An intermediary that converts an HTTP/2 request to HTTP/1.1 MUST create a Host header field if one is not present in a request by copying the value of the ":authority" pseudo-header field. > If the remote user connects to the frontend with HTTP/2 everything is > correct (I assume that is because the :authority pseudo header was already > in there). > > If I remove "proto h2" from the backend, everything is OK too (because in > this case :authority is converted to a Host header when connecting to the > backend). > > Is there something I can do to fix this? Is this a bug (either from > haproxy or Jetty) ? I suspect it's not a bug but a config decision on Jetty's side. Particularly, a few years ago some vulnerabilities were exposed in H2 agents not strictly following the spec regarding host vs :authority, making combinations of such agents rather "fun". It's very possible that some implementations have decided to restrict the fallbacks by default and to only rely on :authority unless configured to look at Host when missing, according to the spec. If required there is a hack that can be used to force an authority. If you force an authority in your request it should normally work: http-request set-uri https://%[req.hdr(host)]%[pathq] But it would be cleaner to make sure the server accepts a request that only has a host and no authority. Regards, Willy
Host header not copied to :authority when using a proto h2 backend
Hello I am using haproxy 2.4.18 with a frontend configured with alpn h2,http/1.1 The problem that I am facing is that if I add "proto h2" to the backends, when a remote client connects to the frontend using HTTP 1.1, the request is sent to the backend server without copying the "Host" header to the ":authority" pseudo-header. This causes the application server (Jetty in my case) to report the backend server numeric IP address as the server name instead of reporting the web domain (which is in the Host header) as one would expect. The URI reported at the application level is also wrong (numeric instead of the domain) for the same reason I suppose. If the remote user connects to the frontend with HTTP/2 everything is correct (I assume that is because the :authority pseudo header was already in there). If I remove "proto h2" from the backend, everything is OK too (because in this case :authority is converted to a Host header when connecting to the backend). Is there something I can do to fix this? Is this a bug (either from haproxy or Jetty) ? Thanks a lot for your help, Óscar
MQTT mqtt_field_value for CONNACK messages
I'm using HAProxy as a proxy for MQTT connections, and I'm looking to
add data from the MQTT CONNACK message (from the server) to my logs.
I have tried the following, however the result is empty ("")
tcp-response inspect-delay 5s
tcp-response content set-var(txn.pv)
res.payload(0,0),mqtt_field_value(connack,protocol_version)
log-format "%{+Q}[var(txn.pv)]"
I have been able to successfully log data from the CONNECT message by
following the examples given in the documentation.
Does anyone have a solution for this?
Many thanks,
Noureddine
AND OR priority when forming conditions
Hello everyone,
I have been using Haproxy for years but I still have trouble
understanding this part of the documentation:
7.2. Using ACLs to form conditions
A condition is formed as a disjunctive form:
[!]acl1 [!]acl2 ... [!]acln { or [!]acl1 [!]acl2 ... [!]acln } ...
first it does not work "as is" if i try something like that:
tcp-request connection reject if { or blacklist_manual tor_ips } !whitelist
it leads to : error detected in frontend 'http_all' while parsing 'if'
condition : unknown fetch method 'or' in ACL expression 'or'.
tcp-request connection reject if { blacklist_manual || tor_ips }
!whitelist does not work eather.
At the end i write something like that:
tcp-request connection reject if blacklist_manual !whitelist || tor_ips
!whitelist
It works but i'm still uncomfortable as i'm not really sure if it's
treated like this : (blacklist_manual !whitelist) || (tor_ips !whitelist)
The documentation on this topic should perhaps be improved, with more
examples with mixed AND/OR. Explain the priority for this type of
operators, and explain what the {or [ !]acl1 [ !]acl2 ... [!]acln }
really means
Thanks !
Re: How to test HTTP/3 on version 2.8?
```
global
log /dev/log local0 info
pidfile /var/run/haproxy.pid
stats socket /var/run/haproxy/api.sock mode 660 level admin
stats timeout 30s
ssl-load-extra-del-ext
ssl-load-extra-files key
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 10m
timeout client 10m
timeout server 10m
frontend mysite
bind :8000
bind :443 ssl crt /home/aidai/haproxy/server.crt alpn h2
# enables HTTP/3 over QUIC
bind quic4@:443 ssl crt /home/aidai/haproxy/server.crt alpn h3
# Redirects to HTTPS
http-request redirect scheme https unless { ssl_fc }
# Switches to the QUIC protocol
http-response set-header alt-svc "h3=\":443\";ma=2592000;"
default_backend webservers
backend webservers
balance roundrobin
server web1 127.0.0.1:8080 check maxconn 30
frontend stats
bind *:8404
stats enable
stats uri /stats
http-request use-service prometheus-exporter if { path /metrics }
stats refresh 10s
stats admin if LOCALHOST
```
and run haproxy with command `haproxy -f haproxy.cfg -d`
Willy Tarreau 于2023年2月24日周五 16:55写道：
> On Fri, Feb 24, 2023 at 04:52:23PM +0800, AiDai wrote:
> > I tried to test outside of Docker, but I encountered the same problem.
> > ```
> > curl --http3-only https://0.0.0.0:443 -k -v
> > * Trying 0.0.0.0:443...
> > * ipv4 connect timeout after 30ms, move on!
> > * Failed to connect to 0.0.0.0 port 443 after 300357 ms: Timeout was
> reached
> > * Closing connection 0
> > curl: (28) Failed to connect to 0.0.0.0 port 443 after 300357 ms: Timeout
> > was reached
> > ```
>
> You should really share your config, as it's very likely that something
> subtle is missing somewhere.
>
> Willy
>
Re: How to test HTTP/3 on version 2.8?
On Fri, Feb 24, 2023 at 04:52:23PM +0800, AiDai wrote: > I tried to test outside of Docker, but I encountered the same problem. > ``` > curl --http3-only https://0.0.0.0:443 -k -v > * Trying 0.0.0.0:443... > * ipv4 connect timeout after 30ms, move on! > * Failed to connect to 0.0.0.0 port 443 after 300357 ms: Timeout was reached > * Closing connection 0 > curl: (28) Failed to connect to 0.0.0.0 port 443 after 300357 ms: Timeout > was reached > ``` You should really share your config, as it's very likely that something subtle is missing somewhere. Willy
Re: How to test HTTP/3 on version 2.8?
I tried to test outside of Docker, but I encountered the same problem. ``` curl --http3-only https://0.0.0.0:443 -k -v * Trying 0.0.0.0:443... * ipv4 connect timeout after 30ms, move on! * Failed to connect to 0.0.0.0 port 443 after 300357 ms: Timeout was reached * Closing connection 0 curl: (28) Failed to connect to 0.0.0.0 port 443 after 300357 ms: Timeout was reached ``` ``` curl --http3 https://0.0.0.0:443 -k -v * Trying 0.0.0.0:443... * Trying 0.0.0.0:443... * Connected to 0.0.0.0 (127.0.0.1) port 443 (#0) * ALPN: offers http/1.1 * TLSv1.2 (OUT), TLS handshake, Client hello (1): * TLSv1.2 (IN), TLS handshake, Server hello (2): * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): * TLSv1.3 (IN), TLS handshake, Certificate (11): * TLSv1.3 (IN), TLS handshake, CERT verify (15): * TLSv1.3 (IN), TLS handshake, Finished (20): * TLSv1.3 (OUT), TLS handshake, Finished (20): * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 * ALPN: server did not agree on a protocol. Uses default. * Server certificate: * subject: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd * start date: Feb 22 12:12:22 2023 GMT * expire date: Feb 22 12:12:22 2024 GMT * issuer: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd * SSL certificate verify result: self signed certificate (18), continuing anyway. * using HTTP/1.x > GET / HTTP/1.1 > Host: 0.0.0.0 > User-Agent: curl/8.0.0-DEV > Accept: */* > * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): * old SSL session ID is stale, removing * HTTP 1.0, assume close after body < HTTP/1.0 200 OK < server: SimpleHTTP/0.6 Python/3.7.8 < date: Fri, 24 Feb 2023 08:39:47 GMT < content-type: text/html; charset=utf-8 < content-length: 5041 < alt-svc: h3=":443";ma=2592000; * HTTP/1.0 connection set to keep alive < connection: keep-alive ``` Does the output ALPN: offers http/1.1 indicate that haproxy HTTP/3 is not enabled? Willy Tarreau 于2023年2月24日周五 16:25写道： > On Fri, Feb 24, 2023 at 04:22:14PM +0800, AiDai wrote: > > Thanks, but it doesn't work. > > ``` > > curl https://0.0.0.0:443 -k -v --http3-only > > * Trying 0.0.0.0:443... > > * ipv4 connect timeout after 30ms, move on! > > * Failed to connect to 0.0.0.0 port 443 after 300905 ms: Timeout was > reached > > * Closing connection 0 > > curl: (28) Failed to connect to 0.0.0.0 port 443 after 300905 ms: Timeout > > was reached. > > ``` > > However, I no longer receive a "connection to 0.0.0.0:443 refused" > error. I > > suspect that there may be additional settings that need to be configured. > > Maybe. You should really test outside of docker. Docker conversion should > only be the final step of any test, once everything is proven to work fine, > because otherwise it's undebuggable and complicates troubleshooting like > crazy as you can see here :-/ > > Willy >
