Request for feedback: Add support for txn args as arguments in converter "bytes"
Hey folks I am writing to gather feedback on an idea before doing the implementation. Per the documentation, converter "bytes" accepts integer values as arguments, but not txn args. i.e. ,bytes(2,8) will work but ,bytes(txn.start_idx,txn.length) will not work. For our use case, we need to parse some binary data (a cookie) to extract some info in haproxy. However, the bytes that need to be extracted are not fixed and will depend on the request. We could use simple arithmetic to determine the index/length for bytes to be extracted and store them in txn args. These txn args can then be used with converter "bytes". I can see that the converter "sub" already supports txn args as arguments. I have successfully validated the proposed idea with an implementation following the same pattern as "sub". Let me know what you think. If there are no concerns, I can send a patch. Thanks Lokesh
Re: [ANNOUNCE] haproxy-2.9-dev4
On Fri, Aug 25, 2023 at 08:52:00PM +0200, Aleksandar Lazic wrote: > Looks like that "Reverse HTTP Transport" will be only possible with H2 & H3 > for now, right. This looks then to me that quic + H3 will be implemented > also for server as "proto h3", right? No, it's only H2 for now. And indeed server-side h3 will be needed before this can be ported to H3. And I don't even know if it's possible to reverse a QUIC connection (I wouldn't want to give Fred a heart attack). However it could combine well with something I wanted to do long ago in a limited way: I wanted to support prepending an HTTP/1 CONNECT header to outgoing TCP connections so that it would be possible to encapsulate outgoing TCP connections into HTTP. Nowadays with protocol muxes it wouldn't work that way anymore, but it would be great if we could upgrade TCP to HTTP CONNECT in H1/H2/H3, because combined with the stuff above, this could even allow to forward TCP through two layers of reverse-H2! Imagine SSHing home from outside by connecting to a public gateway port, or exposing some MQTT services, or accessing some of your phone's applications locally via an always-on reverse-http connection. > Will HAProxy be the first one which will have this or is there anybody else > which have also implemented this into there SW? I don't know. I saw products do that over HTTP/1 20+ years ago. It was absolutely horrible and unreliable. A user clicking stop during a download was sufficient to either break the HTTP connection or have the gateway drain it. H2 offers more possibilities and several of us who discussed about some possibilities during various workshops evocated such future opportunities, without knowing if anyone would have the guts and time to really start it. But it's still possible that in parallel others have already done this, even in proprietary products, because the concept looks quite obvious when you look at the protocol, it's "just" quite tricky to do given all the implicit stuff that characterizes a client or server connection in a gateway! Cheers, Willy
Re: [ANNOUNCE] haproxy-2.9-dev4
Hi. On 2023-08-25 (Fr.) 19:35, Willy Tarreau wrote: Hi, HAProxy 2.9-dev4 was released on 2023/08/25. It added 59 new commits after version 2.9-dev3. Some interesting new stuff continues to arrive in this version: [snipp] - reverse HTTP: see below for a complete description. I hope it will answer Alex's question :-) Thank you :-) - xxhash was updated to 0.8.2 (we were on 0.8.1) because it fixes a build issue on ppc64le. - various doc/regtest/CI updates as usual. Now, regarding reverse HTTP: that's a feature that we've been repeatedly asked for over the last decade, constantly responding "not possible yet". But with the flexibility of the current architecture, it appeared that there was no more big show-stopper and it was about time to respond to this demand. What is this ? The principle is to permit a server to establish a connection to haproxy, then to switch the connection direction on both sides, so that haproxy can send requests to that server. There was a trend around this 20 years ago on HTTP/1 and it didn't work well, to be honest. And we were counting on H2 to do that because it allows to multiplex streams over a connection and to reset a stream without breaking a connection. [snipp good explanation] Looks like that "Reverse HTTP Transport" will be only possible with H2 & H3 for now, right. This looks then to me that quic + H3 will be implemented also for server as "proto h3", right? Will HAProxy be the first one which will have this or is there anybody else which have also implemented this into there SW? Regards Alex
[ANNOUNCE] haproxy-2.9-dev4
Hi, HAProxy 2.9-dev4 was released on 2023/08/25. It added 59 new commits after version 2.9-dev3. Some interesting new stuff continues to arrive in this version: - maps: the set-map action (and equivalent Lua calls) used to update an entry in O(N) for N elements in a map, due to the original design focusing on limiting memory usage. Nowadays despite the warnings in the doc, it appears that more and more users are relying on set-map on the traffic path in Lua or HTTP actions and are sometimes reporting hard-to diagnose CPU usage issues which end up being caused just by this. The reference lookup code is now in O(log(N)) and the propagation in O(1) so this will be much better for these special use cases. The memory increased from 72 to 120 bytes per map entry (to which about as much was already added for each instanciation), so that's reasonable, and will later be shrunk again. In addition, lookups in empty maps used to sollicit the LRU cache anyway, which could represent up to about 4-5% CPU in tests. That's too bad, considering we know the map is empty, hence the cache as well, so now the lookup is avoided on empty maps and acls. - when idle connections were reworked to support SNI, a side effect of replacing the list with a tree was that they were now put back at the end of the list instead of at the head, hence they were all used in a round-robin fashion, something which is not that great when thinking about purging excess connections nor when trying to concentrate most of the traffic on few connections (e.g. window sizes will not necessarily increase etc). This was changed so that the previous behavior is restored and the recently used connections packed together at the head again, so that we have the hottest and most reliable ones at the head and the least trusted ones at the tail. This allows the purge mechanism to kill from the tail and preserve the most recently used ones. - limited-quic: now in order to make sure not to fool users, when building with SSL library which does NOT support QUIC (i.e. OpenSSL), "quic" bindings will be properly rejected unless the "limited-quic" is specified in the config (and it's suggested in the error message). Previously it would only silently ignore them, resulting in a non-working config, causing confusion to those who copy-paste configs without being aware of this. Also a warning is now emitted in this mode when "allow-0rtt" is specified, as the "limited-quic" compatibility layer doesn't support it. - reverse HTTP: see below for a complete description. I hope it will answer Alex's question :-) - xxhash was updated to 0.8.2 (we were on 0.8.1) because it fixes a build issue on ppc64le. - various doc/regtest/CI updates as usual. Now, regarding reverse HTTP: that's a feature that we've been repeatedly asked for over the last decade, constantly responding "not possible yet". But with the flexibility of the current architecture, it appeared that there was no more big show-stopper and it was about time to respond to this demand. What is this ? The principle is to permit a server to establish a connection to haproxy, then to switch the connection direction on both sides, so that haproxy can send requests to that server. There was a trend around this 20 years ago on HTTP/1 and it didn't work well, to be honest. And we were counting on H2 to do that because it allows to multiplex streams over a connection and to reset a stream without breaking a connection. There are 4 use cases I'm currently aware of, though others might be creative: - isolation: a server in a purely outgoing DMZ, connects to the edge load balancer and receives requests from there. There's zero incoming connection to that DMZ. Some security environments require this (not that I fully agree with this, to be honest). - work around painful NAT: mobile developers who want to test their applications on their smartphone often have to either push on a public dev server, or hack around the local network's wifi to permit their phone to connect directly to the dev PC. Here it can be much simpler, their PC connects to a public gateway, registers there and instantly receives the traffic for the configured host name and delivers it to the application running locally in debug mode with traces etc. A similar use case consists in working around the difficulty to set up port- forwarding on some home internet accesses, here you can expose your internal application directly outside via a public gateway again, using exclusively an outgoing connection. Exactly the same can be done with containers: instead of having to know what ports to NAT, it can be convenient to let the server in the container directly register to the external gateway. I'll soon try to setup one on a public server so
Re: [PATCH] MEDIUM: sample: Implement sample fetch for arbitrary PROXY protocol v2 TLV values
Hi Alexander, On Fri, Aug 25, 2023 at 09:34:08AM +, Stephan, Alexander wrote: > Hi Willy, > > Did you already have the chance to take a look at the updated patches? > No hurry though, I just wanted to make sure that the message didn't get lost. Not yet, I'm still burried under annoying bugs for now. I was just telling a coworker that I intend to review your work early next week. I'm fairly confident but it requires time. Thanks! Willy
RE: [PATCH] MEDIUM: sample: Implement sample fetch for arbitrary PROXY protocol v2 TLV values
Hi Willy, Did you already have the chance to take a look at the updated patches? No hurry though, I just wanted to make sure that the message didn't get lost. As mentioned, I am aware that sending individual patches is better in the common case. If that is a problem here, please just let me know. Best, Alexander
Re: [PATCH 1/1] CI: fedora: fix "dnf" invocation syntax
On Fri, Aug 25, 2023 at 08:20:24AM +0200, Ilya Shipitsin wrote: > In 39e205302e8afb68d954e17944fad5f8c1587287 I erroneously used > "dnf git" instead of "dnf install" > --- > .github/workflows/fedora-rawhide.yml | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/.github/workflows/fedora-rawhide.yml > b/.github/workflows/fedora-rawhide.yml > index bf8ad3d72..aa3abc585 100644 > --- a/.github/workflows/fedora-rawhide.yml > +++ b/.github/workflows/fedora-rawhide.yml > @@ -20,7 +20,7 @@ jobs: > - uses: actions/checkout@v3 > - name: Install dependencies >run: | > -dnf -y git install pcre-devel zlib-devel pcre2-devel 'perl(FindBin)' > perl-IPC-Cmd 'perl(File::Copy)' 'perl(File::Compare)' lua-devel socat > findutils systemd-devel clang > +dnf -y install git pcre-devel zlib-devel pcre2-devel 'perl(FindBin)' > perl-IPC-Cmd 'perl(File::Copy)' 'perl(File::Compare)' lua-devel socat > findutils systemd-devel clang > - name: Install VTest >run: scripts/build-vtest.sh > - name: Install QuicTLS Thanks, merged. -- William Lallemand
[PATCH 0/1] CI: fedora: fix dnf invocation
"dnf git" was used instead of "dnf install" Ilya Shipitsin (1): CI: fedora: fix "dnf" invocation syntax .github/workflows/fedora-rawhide.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- 2.35.3.windows.1
[PATCH 1/1] CI: fedora: fix "dnf" invocation syntax
In 39e205302e8afb68d954e17944fad5f8c1587287 I erroneously used "dnf git" instead of "dnf install" --- .github/workflows/fedora-rawhide.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/fedora-rawhide.yml b/.github/workflows/fedora-rawhide.yml index bf8ad3d72..aa3abc585 100644 --- a/.github/workflows/fedora-rawhide.yml +++ b/.github/workflows/fedora-rawhide.yml @@ -20,7 +20,7 @@ jobs: - uses: actions/checkout@v3 - name: Install dependencies run: | -dnf -y git install pcre-devel zlib-devel pcre2-devel 'perl(FindBin)' perl-IPC-Cmd 'perl(File::Copy)' 'perl(File::Compare)' lua-devel socat findutils systemd-devel clang +dnf -y install git pcre-devel zlib-devel pcre2-devel 'perl(FindBin)' perl-IPC-Cmd 'perl(File::Copy)' 'perl(File::Compare)' lua-devel socat findutils systemd-devel clang - name: Install VTest run: scripts/build-vtest.sh - name: Install QuicTLS -- 2.35.3.windows.1
