Re: gitlab server behind haproxy never switches to http/3
On 4/27/23 20:59, Tristan wrote: Hmm. The web server is on the local gigabit LAN with the client. Would that give TCP a significant enough boost that it could beat UDP? Hard to say; in our case it seemed more random than actually driven by anything remotely close to clear. I'm merely quoting official word on it, but have yet to try and dig the actual source code for how it happens. The only platform I know gets reliable QUIC enabling is CF, so maybe someone from their side could chime in if they ever see this. I know the NICs in the server do TCP handling/acceleration. It is likely that the NIC in my desktop also does it. A packet capture shows that the server NIC also handles UDP (outbound packets have a bad checksum because it is actually calculated by the NIC) ... but maybe the TCP handling is a lot more efficient than the UDP handling. It would receive more attention from hardware engineers because the majority of IP traffic in the wild is TCP. Thanks for your insights. I have not delved into quic as deeply as you have. I know just enough about it that I can get it working on haproxy and I can ask annoying questions. :) Thanks, Shawn
Re: gitlab server behind haproxy never switches to http/3
I did figure out that ufw was not allowing udp/443. So it turns out that wasn't working for any website on that install. Ah yep, the fact that allow 443 implies allow tcp/443 and you need to explicitly allow udp/443 is yet another "quirky" thing (to be polite...) But even after allowing udp/443, it is still not switching. If I enter the URL into https://http3check.net/ it does say that http3 works. Yup that sounds about right Hmm. The web server is on the local gigabit LAN with the client. Would that give TCP a significant enough boost that it could beat UDP? Hard to say; in our case it seemed more random than actually driven by anything remotely close to clear. I'm merely quoting official word on it, but have yet to try and dig the actual source code for how it happens. The only platform I know gets reliable QUIC enabling is CF, so maybe someone from their side could chime in if they ever see this. I have chrome, not chromium. Substituting /opt/google/chrome/chrome for chromium and running it with those options, it DOES do http/3. The lightning bolt is orange instead of blue. Then yep, you're in the same boat as we were. It switched for no reason one day. Even trying HTTPS/SVCB DNS records did nothing for us until it "magically" decided to use H3. Tristan
Re: gitlab server behind haproxy never switches to http/3
I did figure out that ufw was not allowing udp/443. So it turns out that wasn't working for any website on that install. I have another install in AWS that IS working, and it turns out that when I was seeing the green lightning bolt in firefox, it was one of those websites, not the ones in my local install. Sometimes I forget which install covers specific websites. But even after allowing udp/443, it is still not switching. If I enter the URL into https://http3check.net/ it does say that http3 works. On 4/27/23 20:26, Tristan wrote: As far as I know, the main way it happens is that the browser: - races H2 and H3 and picks the fastest (then remember it) - retries on H2 in case of H3 issue (then remember it) Hmm. The web server is on the local gigabit LAN with the client. Would that give TCP a significant enough boost that it could beat UDP? What I learned from forcing quic (see below) seems to support this notion. > You can try something like that to force it to use H3 and reveal > whatever issue it might be having: > chromium-browser --enable-quic > --origin-to-force-quic-on=your-gitlab-host.com:443 I have chrome, not chromium. Substituting /opt/google/chrome/chrome for chromium and running it with those options, it DOES do http/3. The lightning bolt is orange instead of blue. Thanks, Shawn
Re: gitlab server behind haproxy never switches to http/3
One of the websites I have behind it is my gitlab server. That is
always http/2, it never switches to http/3.
Does anyone know why that happens, and whether there is anything I can
do about it? The alt-svc header IS received by the browser.
Browsers unfortunately do not give much information in such cases. Your
best bet is going to {chrome, edge}://net-export/ and recording a session.
Sometimes the browser will be kind enough to acknowledge it with a
reason there, but usually it won't (which is quite infuriating, if
someone working with browser vendors reads this).
Similarly to you, on our site we had the homepage (ie /) refuse to ever
use HTTP3 for the longest time, while all other pages would. Then it
suddenly worked. And then it didn't anymore. etc. Never knew why, never
showed up in net-export nor anywhere else for that matter.
As far as I know, the main way it happens is that the browser:
- races H2 and H3 and picks the fastest (then remember it)
- retries on H2 in case of H3 issue (then remember it)
And how long they remember is not document visibly anywhere that I could
find either (except when it shows up in the net-export, then it says
until when it won't try again, but most of the time it doesn't show up,
again).
You can try something like that to force it to use H3 and reveal
whatever issue it might be having:
chromium-browser --enable-quic
--origin-to-force-quic-on=your-gitlab-host.com:443
And see if anything's broken which would explain why your browser then
refuses to ever use H3 with it.
If you see something then best is to raise an issue; my guess is that
Amaury or Fred will want traces to be able to dig into it (like so
https://github.com/haproxy/haproxy/issues/2004#issuecomment-1398607279)
but that's for them to request if and when they judge it necessary.
Tristan
PS: I see you already do, but for others reading this, you definitely
want 2.7.7 or a 2.8 base after 7b516d3732aab5999fc8f4a3037a4317d35d3618
if you're playing with H3/QUIC. The recent few commits make a
significant difference in compatibility.
gitlab server behind haproxy never switches to http/3
I have haproxy installed, doing http/3. The http/3 is working for
almost everything. I build it from the main branch and update it fairly
often.
One of the websites I have behind it is my gitlab server. That is
always http/2, it never switches to http/3.
Does anyone know why that happens, and whether there is anything I can
do about it? The alt-svc header IS received by the browser.
elyograg@smeagol:/storage0/build/haproxy$ haproxy -vv
HAProxy version 2.8-dev8-d2f61d-40 2023/04/27 - https://haproxy.org/
Status: development branch - not safe for use in production.
Known bugs: https://github.com/haproxy/haproxy/issues?q=is:issue+is:open
Running on: Linux 6.1.0-1009-oem #9-Ubuntu SMP PREEMPT_DYNAMIC Fri Mar
31 09:59:10 UTC 2023 x86_64
Build options :
TARGET = linux-glibc
CPU = native
CC = cc
CFLAGS = -O2 -march=native -g -Wall -Wextra -Wundef
-Wdeclaration-after-statement -Wfatal-errors -Wtype-limits
-Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond
-Wnull-dereference -fwrapv -Wno-address-of-packed-member
-Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered
-Wno-missing-field-initializers -Wno-cast-function-type
-Wno-string-plus-int -Wno-atomic-alignment
OPTIONS = USE_OPENSSL=1 USE_ZLIB=1 USE_SYSTEMD=1 USE_QUIC=1
USE_PCRE2_JIT=1
DEBUG = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS
Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY
+CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE
-LIBATOMIC +LIBCRYPT +LINUX_SPLICE +LINUX_TPROXY -LUA -MATH
-MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL
-OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL
-PROCCTL -PROMEX -PTHREAD_EMULATION +QUIC +RT +SHM_OPEN -SLZ +SSL
-STATIC_PCRE -STATIC_PCRE2 +SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY
-WURFL +ZLIB
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256,
default=48).
Built with OpenSSL version : OpenSSL 3.1.0+quic 14 Mar 2023
Running on OpenSSL version : OpenSSL 3.1.0+quic 14 Mar 2023
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
OpenSSL providers loaded : default
Built with network namespace support.
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT
IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.39 2021-10-29
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 11.3.0
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as cannot be specified using 'proto' keyword)
quic : mode=HTTP side=FE mux=QUIC flags=HTX|NO_UPG|FRAMED
h2 : mode=HTTP side=FE|BE mux=H2flags=HTX|HOL_RISK|NO_UPG
fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG
: mode=HTTP side=FE|BE mux=H1flags=HTX
h1 : mode=HTTP side=FE|BE mux=H1flags=HTX|NO_UPG
: mode=TCP side=FE|BE mux=PASS flags=
none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG
Available services : none
Available filters :
[BWLIM] bwlim-in
[BWLIM] bwlim-out
[CACHE] cache
[COMP] compression
[FCGI] fcgi-app
[SPOE] spoe
[TRACE] trace
Thanks,
Shawn
Re: [ANNOUNCE] haproxy-2.7.7
On Thu, Apr 27, 2023 at 04:59:24PM +0200, Christopher Faulet wrote: > Hi, > > HAProxy 2.7.7 was released on 2023/04/27. It added 163 new commits > after version 2.7.6. > > This release is pretty huge. In one month, the QUIC team achieved an amazing > work to improve the stack and make it more stable. A big thanks to Tristan > for his priceless help. More than half of commits concern the QUIC stack. It > is hard to sum up all changes. Many bugs were fixed, the most visible are: > > * The Congestion algorithms state was shared between connections instead of >being private. > > * HTTP/1.0 responses with an unknown content length and finished on close >were not properly handled. It was considered as an early close by the >QUIC multiplexer, leading to a RESET_STREAM emission. > > * The streams fairness was improved to prevent timeouts. A stream sending a >large object could block other streams. With a small client timeout, >blocked streams could be aborted via a RESET_STREAM. > > * Some contradictions in code could lead to very long loops sending empty >packets (PADDING only packets). One visible effect was a very low >throughput performance when the client serialized its requests. > > * The control window in congestion algorithms could be zero because of a >wrong calculation and could lead to a SIGFPE crash. > > * Padding was missing in very short probe packets > > * Possible memory leaks were fixed > > And of course, some improvements were brought. The main one is the support > of the thread loadbalancing on accept. A series of changes allowed to use > the default mechanism to accept and handle connections. The less loaded > thread is now selected, improving the global performance of the QUIC > stack. In addition, a timer was added to delay the acknowledgments. > > Recent refactoring about the stream-connector layer introduced a regression > since the 2.7.4. The read timer was no longer rearmed if the end of the > message was reached. This change was introduced to avoid server timeouts > when the server replies before the end of the request. But it revealed > several bugs, some was fixed, but some others pretty are hard to fix without > changing some internals. It is too sensitive for a stable version. Thus for > now we decided to revert this change, waiting for a better solution. Note > that it is not a big deal because we only restore a behavior that has been > there for ages. > > On soft-stop or reload, idle DNS session are now killed. Since the 2.7.5, > these sessions were no longer killed, preventing the process to finish. In > addition, we now force the connect timeout for the DNS resolution. The > "resolve" timeout is used to set its value. Have no connect timeout was an > issue for resolution over TCP. Connection failures might take quite long to > report, leading to an excess of unusable DNS sessions in connecting > state. It was especially visible on soft-stop because this prevented the > process to quickly exit. Still on the DNS, errors are now properly handled > when a response is consumed. This was an issue for truncated responses > followed by an abort. The applet could ignore the abort and loop waiting for > more data until a timeout is triggered. A similar issue was fixed in the > syslog applet. > > Several bugs in lua part were fixed. First, except for lua tasks, it is no > longer possible to register functions at runtime. It was clearly stated in > the documentation, but nothing forbidden it in the code. An error is now > triggered if this happens, preventing potential segfaults. Memory leaks on > references were fixed and the lua locking was simplified to be re-entrant to > prevent deadlocks. > > Aurélien fixed several issues on the servers management. The "visible" > server list consistency was fixed. It was possible, at least in theory, to > access an invalid server if several dynamic server deletions were performed > while the list was accessed. For instance it might happen when the server > list was dumped in the stats. He also fixed wrong report for tracking > servers leaving drain state. Finally, he centralized proxy and server stats > updates on server state transition to be sure to not miss an update on some > transitions. > > The issues that were occasionally met around the use of malloc_trim() that > had been addressed in 2.8 were finally backported after one month of > exposure in 2.8. The issue was that not only malloc_trim() could still > sometimes be used when jemalloc (or any other allocator) was used, but our > attempts at plugging these special cases didn't work when linking with > external libs that also explicitly call it. In the end, the opposite was > done: we redefine our own version of malloc_trim(), which contains the tests > for the presence of an alternate lib, and call the original if the allocator > comes from libc, or call the equivalent function from other allocators. This > way external libs that would use
[ANNOUNCE] haproxy-2.7.7
Hi, HAProxy 2.7.7 was released on 2023/04/27. It added 163 new commits after version 2.7.6. This release is pretty huge. In one month, the QUIC team achieved an amazing work to improve the stack and make it more stable. A big thanks to Tristan for his priceless help. More than half of commits concern the QUIC stack. It is hard to sum up all changes. Many bugs were fixed, the most visible are: * The Congestion algorithms state was shared between connections instead of being private. * HTTP/1.0 responses with an unknown content length and finished on close were not properly handled. It was considered as an early close by the QUIC multiplexer, leading to a RESET_STREAM emission. * The streams fairness was improved to prevent timeouts. A stream sending a large object could block other streams. With a small client timeout, blocked streams could be aborted via a RESET_STREAM. * Some contradictions in code could lead to very long loops sending empty packets (PADDING only packets). One visible effect was a very low throughput performance when the client serialized its requests. * The control window in congestion algorithms could be zero because of a wrong calculation and could lead to a SIGFPE crash. * Padding was missing in very short probe packets * Possible memory leaks were fixed And of course, some improvements were brought. The main one is the support of the thread loadbalancing on accept. A series of changes allowed to use the default mechanism to accept and handle connections. The less loaded thread is now selected, improving the global performance of the QUIC stack. In addition, a timer was added to delay the acknowledgments. Recent refactoring about the stream-connector layer introduced a regression since the 2.7.4. The read timer was no longer rearmed if the end of the message was reached. This change was introduced to avoid server timeouts when the server replies before the end of the request. But it revealed several bugs, some was fixed, but some others pretty are hard to fix without changing some internals. It is too sensitive for a stable version. Thus for now we decided to revert this change, waiting for a better solution. Note that it is not a big deal because we only restore a behavior that has been there for ages. On soft-stop or reload, idle DNS session are now killed. Since the 2.7.5, these sessions were no longer killed, preventing the process to finish. In addition, we now force the connect timeout for the DNS resolution. The "resolve" timeout is used to set its value. Have no connect timeout was an issue for resolution over TCP. Connection failures might take quite long to report, leading to an excess of unusable DNS sessions in connecting state. It was especially visible on soft-stop because this prevented the process to quickly exit. Still on the DNS, errors are now properly handled when a response is consumed. This was an issue for truncated responses followed by an abort. The applet could ignore the abort and loop waiting for more data until a timeout is triggered. A similar issue was fixed in the syslog applet. Several bugs in lua part were fixed. First, except for lua tasks, it is no longer possible to register functions at runtime. It was clearly stated in the documentation, but nothing forbidden it in the code. An error is now triggered if this happens, preventing potential segfaults. Memory leaks on references were fixed and the lua locking was simplified to be re-entrant to prevent deadlocks. Aurélien fixed several issues on the servers management. The "visible" server list consistency was fixed. It was possible, at least in theory, to access an invalid server if several dynamic server deletions were performed while the list was accessed. For instance it might happen when the server list was dumped in the stats. He also fixed wrong report for tracking servers leaving drain state. Finally, he centralized proxy and server stats updates on server state transition to be sure to not miss an update on some transitions. The issues that were occasionally met around the use of malloc_trim() that had been addressed in 2.8 were finally backported after one month of exposure in 2.8. The issue was that not only malloc_trim() could still sometimes be used when jemalloc (or any other allocator) was used, but our attempts at plugging these special cases didn't work when linking with external libs that also explicitly call it. In the end, the opposite was done: we redefine our own version of malloc_trim(), which contains the tests for the presence of an alternate lib, and call the original if the allocator comes from libc, or call the equivalent function from other allocators. This way external libs that would use it are safe as well. The mixed library version detection in dlopen() was still a bit sensitive, and could sometimes detect anomalies related to an external lib depending on libcrypto but not libssl for example, as well as libs that were
https://www.haproxy.org/
-- Hi Dear, I want to post here( https://www.haproxy.org/ ) let me know the price for each post. I am waiting for your reply... Thank You Anila
Re: Interface?
Hello, how ist it with: https://github.com/hap-wi/roxy-wi Im using also pfsense firewall, which has also a gui greets Stefan Am 27.04.23 um 04:32 schrieb Jeremy Hansen: Just curious if there’s any useful web interfaces for managing HAProxy configs? Thanks -jeremy
