Re: Peers using heavily single cpu core
On Fri, Mar 11, 2022 at 10:19:09PM +0100, Maciej Zdeb wrote: > Hi Willy, > > Thank you for such useful info! I've checked the worst HAProxy nodes and on > every such node all outgoing peers connections are run on the same thread: (...) Indeed. I was pretty sure we were setting them to any thread on creation but maybe I'm wrong, I'll have to recheck. > On one node I was able to rebalance it, but on the node above (and other > nodes) I'm not able to shutdown the sessions: (...) > echo "shutdown session 0x7f0aa402e2c0" | socat > unix-connect:/var/run/haproxy.sock stdio > > echo "show sess 0x7f0aa402e2c0" | socat unix-connect:/var/run/haproxy.sock > stdio > 0x7f0aa402e2c0: [11/Mar/2022:07:17:02.313221] id=0 proto=? (...) That's not expected, another thing I'll have to check. Thanks for testing. I'll put that in pause for the week-end, though :-) cheers, Willy
[PATCH] REGTESTS: Do not use REQUIRE_VERSION for HAProxy 2.5+
Introduced in:
0657b9338 MINOR: stream: add "last_rule_file" and "last_rule_line" samples
---
reg-tests/log/last_rule.vtc | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/reg-tests/log/last_rule.vtc b/reg-tests/log/last_rule.vtc
index b57251912..e69516654 100644
--- a/reg-tests/log/last_rule.vtc
+++ b/reg-tests/log/last_rule.vtc
@@ -1,7 +1,7 @@
varnishtest "Verify logging of last final rule"
-feature ignore_unknown_macro
-#REQUIRE_VERSION=2.6
+feature cmd "$HAPROXY_PROGRAM -cc 'version_atleast(2.6-dev0)'"
+feature ignore_unknown_macro
server s1 {
rxreq
--
2.35.1
Re: Peers using heavily single cpu core
Hi Willy, Thank you for such useful info! I've checked the worst HAProxy nodes and on every such node all outgoing peers connections are run on the same thread: echo "show fd" | socat unix-connect:/var/run/haproxy.sock stdio | grep "px=" 430 : st=0x010121(cL heopI W:sRa R:srA) tmask=0x40 umask=0x0 owner=0x7f0aa4039250 iocb=0x564c0df95f40(sock_conn_iocb) back=1 cflg=0x10002300 fam=ipv4 lport=22850 rport=1024 px=hap. mux=PASS ctx=0x7f0aa4039450 xprt=RAW 500 : st=0x010121(cL heopI W:sRa R:srA) tmask=0x40 umask=0x0 owner=0x7f0aa403d4e0 iocb=0x564c0df95f40(sock_conn_iocb) back=1 cflg=0x10002300 fam=ipv4 lport=58906 rport=1024 px=hap mux=PASS ctx=0x7f0aa403d6e0 xprt=RAW 501 : st=0x010121(cL heopI W:sRa R:srA) tmask=0x40 umask=0x0 owner=0x7f0aa4041770 iocb=0x564c0df95f40(sock_conn_iocb) back=1 cflg=0x10002300 fam=ipv4 lport=54470 rport=1024 px=hap mux=PASS ctx=0x7f0aa4041970 xprt=RAW 502 : st=0x010121(cL heopI W:sRa R:srA) tmask=0x40 umask=0x0 owner=0x7f0aa4045a00 iocb=0x564c0df95f40(sock_conn_iocb) back=1 cflg=0x10002300 fam=ipv4 lport=48926 rport=1024 px=hap mux=PASS ctx=0x7f0aa4045c00 xprt=RAW 503 : st=0x010121(cL heopI W:sRa R:srA) tmask=0x40 umask=0x0 owner=0x7f0aa4049c90 iocb=0x564c0df95f40(sock_conn_iocb) back=1 cflg=0x10002300 fam=ipv4 lport=21468 rport=1024 px=hap mux=PASS ctx=0x7f0aa4049e90 xprt=RAW echo "show sess" | socat unix-connect:/var/run/haproxy.sock stdio | grep "proto=?" | grep "epoch=0 " | grep 14h 0x7f0aa402e2c0: proto=? ts=00 epoch=0 age=14h53m calls=10724 rate=1 cpu=0 lat=0 rq[f=848202h,i=0,an=00h,rx=5s,wx=,ax=] rp[f=80448202h,i=0,an=00h,rx=,wx=,ax=] s0=[8,204048h,fd=-1,ex=] s1=[8,2000d8h,fd=430,ex=] exp=3s 0x7f0aa402ece0: proto=? ts=00 epoch=0 age=14h53m calls=10724 rate=1 cpu=0 lat=0 rq[f=848202h,i=0,an=00h,rx=5s,wx=,ax=] rp[f=80448202h,i=0,an=00h,rx=,wx=,ax=] s0=[8,204048h,fd=-1,ex=] s1=[8,2000d8h,fd=500,ex=] exp=3s 0x7f0aa402f700: proto=? ts=00 epoch=0 age=14h53m calls=10724 rate=1 cpu=0 lat=0 rq[f=848202h,i=0,an=00h,rx=5s,wx=,ax=] rp[f=80448202h,i=0,an=00h,rx=,wx=,ax=] s0=[8,204048h,fd=-1,ex=] s1=[8,2000d8h,fd=501,ex=] exp=3s 0x7f0aa4030120: proto=? ts=00 epoch=0 age=14h53m calls=10722 rate=0 cpu=0 lat=0 rq[f=848202h,i=0,an=00h,rx=5s,wx=,ax=] rp[f=80448202h,i=0,an=00h,rx=,wx=,ax=] s0=[8,204048h,fd=-1,ex=] s1=[8,2000d8h,fd=502,ex=] exp=2s 0x7f0aa4030b40: proto=? ts=00 epoch=0 age=14h53m calls=10724 rate=1 cpu=0 lat=0 rq[f=848202h,i=0,an=00h,rx=5s,wx=,ax=] rp[f=80448202h,i=0,an=00h,rx=,wx=,ax=] s0=[8,204048h,fd=-1,ex=] s1=[8,2000d8h,fd=503,ex=] exp=3s On one node I was able to rebalance it, but on the node above (and other nodes) I'm not able to shutdown the sessions: echo "show sess 0x7f0aa402e2c0" | socat unix-connect:/var/run/haproxy.sock stdio 0x7f0aa402e2c0: [11/Mar/2022:07:17:02.313221] id=0 proto=? flags=0x6, conn_retries=3, srv_conn=(nil), pend_pos=(nil) waiting=0 epoch=0 frontend=x (id=4294967295 mode=http), listener=? (id=0) backend=x (id=4294967295 mode=http) addr=x:22850 server= (id=0) addr=y:1024 task=0x7f0aa402e7d0 (state=0x00 nice=0 calls=10741 rate=0 exp=1s tmask=0x40 age=14h54m) si[0]=0x7f0aa402e600 (state=EST flags=0x204048 endp0=APPCTX:0x7f0aa402df90 exp= et=0x000 sub=0) si[1]=0x7f0aa402e658 (state=EST flags=0x2000d8 endp1=CS:0x7f0aa4039400 exp= et=0x000 sub=1) app0=0x7f0aa402df90 st0=7 st1=0 st2=0 applet= tmask=0x40 nice=0 calls=206051832 rate=2860 cpu=0 lat=0 co1=0x7f0aa4039250 ctrl=tcpv4 xprt=RAW mux=PASS data=STRM target=PROXY:0x564c0ff53760 flags=0x10003300 fd=430 fd.state=10121 updt=0 fd.tmask=0x40 cs=0x7f0aa4039400 csf=0x8200 ctx=(nil) req=0x7f0aa402e2d0 (f=0x848202 an=0x0 pipe=0 tofwd=-1 total=9489764648) an_exp= rex=4s wex= buf=0x7f0aa402e2d8 data=(nil) o=0 p=0 i=0 size=0 res=0x7f0aa402e330 (f=0x80448202 an=0x0 pipe=0 tofwd=-1 total=9320288685) an_exp= rex= wex= buf=0x7f0aa402e338 data=(nil) o=0 p=0 i=0 size=0 echo "shutdown session 0x7f0aa402e2c0" | socat unix-connect:/var/run/haproxy.sock stdio echo "show sess 0x7f0aa402e2c0" | socat unix-connect:/var/run/haproxy.sock stdio 0x7f0aa402e2c0: [11/Mar/2022:07:17:02.313221] id=0 proto=? flags=0x6, conn_retries=3, srv_conn=(nil), pend_pos=(nil) waiting=0 epoch=0 frontend=x (id=4294967295 mode=http), listener=? (id=0) backend=x (id=4294967295 mode=http) addr=x:22850 server= (id=0) addr=x:1024 task=0x7f0aa402e7d0 (state=0x00 nice=0 calls=10745 rate=1 exp=4s tmask=0x40 age=14h55m) si[0]=0x7f0aa402e600 (state=EST flags=0x204048 endp0=APPCTX:0x7f0aa402df90 exp= et=0x000 sub=0) si[1]=0x7f0aa402e658 (state=EST flags=0x2000d8 endp1=CS:0x7f0aa4039400 exp= et=0x000 sub=1) app0=0x7f0aa402df90 st0=7 st1=0 st2=0 applet= tmask=0x40 nice=0 calls=206100614 rate=3160 cpu=0 lat=0 co1=0x7f0aa4039250 ctrl=tcpv4 xprt=RAW mux=PASS data=STRM target=PROXY:0x564c0ff53760 flags=0x10003300 fd=430 fd.state=10121 updt=0 fd.tmask=0x40 cs=0x7f0aa4039400 csf=0x8200 ctx=(nil) req=0x7f0aa402e2d0 (f=0x848202 an=0x0 pipe=0 tofwd=-1
[PR] fix: Correct data types list
Dear list! Author: Fionera Number of patches: 1 This is an automated relay of the Github pull request: fix: Correct data types list Patch title(s): fix: Correct data types list Link: https://github.com/haproxy/haproxy/pull/1603 Edit locally: wget https://github.com/haproxy/haproxy/pull/1603.patch && vi 1603.patch Apply locally: curl https://github.com/haproxy/haproxy/pull/1603.patch | git am - Description: There is a Typo in the peers-v2.0 Doc Instructions: This github pull request will be closed automatically; patch should be reviewed on the haproxy mailing list (haproxy@formilux.org). Everyone is invited to comment, even the patch's author. Please keep the author and list CCed in replies. Please note that in absence of any response this pull request will be lost.
Re: [PATCH 0/6] 'ist'ify members of struct proxy
Willy, [Dropping Christopher from Cc] On 3/9/22 08:11, Willy Tarreau wrote: As for the second CLEANUP commit: If one of you knows how to fix the Coccinelle patch to detect that specific pattern, I'd appreciate if you could make the necessary changes to ist.cocci. Unfortunately my Coccinelle skills are not good enough. I've already faced this situation where it didn't find fields of a given type inside a structure, and I stopped searching when I figured that by the time I would finally find, I could have replaced 10 times the 3 occurrences I needed. I essentially use Coccinelle as a helpful tool to save me time, and I can't resolve myself to spend more time trying to write unmaintainable scripts that will never be reused. A good source of inspiration are the scripts in the linux kernel, but those are often of an extreme level of complexity and mix python scripting within the patch, resulting in me not understanding anything anymore. But some of them are still human-readable or at least show the syntax hacks you're looking for. Yeah, I've attempted to look into the Coccinelle patches in the Linux kernel sources, but I agree that many of those are very complex :-) Do you happen to know where we could ask for assistance with making the necessary adjustments to the patches? Or perhaps you could ask and include me in Cc, at least people already know you. I'll be happy to further improve the existing Coccinelle patches and to further 'ist'ify the codebase, but would need some handholding to get me started. Best regards Tim Düsterhus
Re: [ANNOUNCE] haproxy-2.6-dev3
Hi Tim, On Fri, Mar 11, 2022 at 07:24:42PM +0100, Tim Düsterhus wrote: > Willy, > > On 3/11/22 19:05, Willy Tarreau wrote: > > So unless anyone has extremely good insights about this or wants to > > share some return of experience from the field about similar issues, > > I'm going to work on this next week, trying to make this backportable > > to 2.5 at least. > > I don't have any insights to share, but as this touches an area that at > least is partly related to: > > https://github.com/haproxy/haproxy/issues/969 Ah indeed! > How is the current status of that one? I didn't even remember about it, there are far too many feature requests already :-( > I'm still interested in having that feature, Of course, I agree. > but it's likely out of reach for my current skill level within > HAProxy :-) I can easily understand. I don't know how to implement it either, it's complicated because streams are not supposed to act on the connection so there's no simple mechanism to do that (which is likely why it has been left that long with no progress). Something needs to be added to pass such a signal down the stack, in a way that remains compatible with the different versions. With HTTP/1 that's done by appending a "connection: close" header field, but I'm not much thrilled at the idea of making the H2 mux have a look at this one. I'll discuss this with Christopher who's still burried in all these intermediary layers, so that we can figure an elegant (hence durable) method to achieve this. Thanks for the reminder! Willy
Re: [ANNOUNCE] haproxy-2.6-dev3
Willy, On 3/11/22 19:05, Willy Tarreau wrote: So unless anyone has extremely good insights about this or wants to share some return of experience from the field about similar issues, I'm going to work on this next week, trying to make this backportable to 2.5 at least. I don't have any insights to share, but as this touches an area that at least is partly related to: https://github.com/haproxy/haproxy/issues/969 How is the current status of that one? I'm still interested in having that feature, but it's likely out of reach for my current skill level within HAProxy :-) Best regards Tim Düsterhus
[ANNOUNCE] haproxy-2.6-dev3
Hi, HAProxy 2.6-dev3 was released on 2022/03/11. It added 119 new commits after version 2.6-dev2. A few issues (~25) were fixed at various places in the code. A few of them will probably warrant new stable versions next week, we'll see; it was found that there's a risk of connection leaks on TCP backends which take time to connect if the client gives up before the end. This could happen with a server whose network access is flaky for example. - dynamic servers got a small change. The syntax was chosen very close to the one from the config file but this raised some concerns about long- term issues since some options are implicit (or work in combination), and while that's inevitable in a two-decade human-readable file format, it's not a good idea at all to maintain such problems with commands that will mostly be set by programs. Typically, setting SSL used to implicitly turn on SSL for health checks, and the "addr" and "port" values would implicitly apply both to "check" and "agent". It will be impossible for a program to try to replicate the internal logic to adjust such settings, so we preferred to make all of them explicit (essentially "ssl" vs "check" part above, as the rest was already correct). The "experimental" status of the option was dropped and the examples updated in the doc, as the mechanism should be stable now. - the HTTP client could loop forever in Lua when reusing an existing client instance for a second POST request due to a transfer counter that was not properly reset - usual bunch of QUIC updates, this time mostly focusing on the outgoing traffic and overall correctness of internal representation. I've been pleased to see that the interop tests start to regularly report full- green status with certain client implementations. It's a bit random due to loss/corruption tests, and the fact that it's all green doesn't mean it's prod-ready yet, but a good indication that things are progressing at a pace that's encouraging for 2.6. - a new global "no-memory-trimming" setting was added; we've seen a report of a user running with tens of GB of RAM and for whom the call to malloc_trim() on reload took so long that it managed to trigger the watchdog. That was a good enough indication that malloc_trim() may hurt more than it helps on certain setups, hence the new option to turn it off. - some debugging assertions were added at various places in the code to try to catch certain classes of bugs more easily. Now the DEBUG_STRICT macro supports a new value (2) to enable checks in the hot path, which means that value 1 now only enables those which do not have any measurable performance impact. As such it was turned on by default, hoping to stop certain bugs earlier. The work of adding more of these at sensitive places continues. - a tiny UDP proxy dedicated to fault injection was added to help testing QUIC. It could be reused for DNS and syslog. For now it only reorders/loses/duplicates packets, which is sufficient for datagram-oriented services. More options should be added (truncation and bit flipping), and if some are interested in having fun playing with that, they're welcome to have a look. - some core cleanups that also aim at limiting the risk of introducing new bugs (e.g. switch to struct ist for a number of config strings). - a new pair of sample fetch functions "last_rule_file" and "last_rule_line" which will locate in the config file which tcp/http rule was the last one to give a final verdict (accept/deny/redirect etc). They can be added to logs via log-format to help figure which rule is causing a 403 for example. If there's interest for this, it should be backportable to 2.5. - the dark mode of the "socket" rows in the stats page was finally fixed! - usual lot of build cleanups, CI and doc updates In addition I'm planning on performing a small change soon in the way the timeouts are handled in H2, and depending on the difficulty, it may or may not be backported: Christian described me a situation where an H2 connection that only gets control traffic will not timeout (this may be between applications using PING frames for a heartbeat for example). In this case when there's no stream, only "show fd" will show what is happening but the timeouts cannot be exploited to force such a connection to be closed after some time as only client/server are used and there's periodic traffic. After discussing with him and Christopher, I got convinced that this is *exactly* the same as what we do in HTTP/1 using "timeout http-request" and "timeout http-keep-alive". Indeed, the former is used to indicate how long we'll wait for a full request on a new connection, and how long we'll let a client send a full request once a new one is started. The later indicates how long we keep the connection open after a
Re: Peers using heavily single cpu core
Hi Maciej, On Fri, Mar 04, 2022 at 01:10:37PM +0100, Maciej Zdeb wrote: > Hi, > > I'm experiencing high CPU usage on a single core, idle drops below 40% > while other cores are at 80% idle. Peers cluster is quite big (12 servers, > each server running 12 threads) and is used for synchronization of 3 > stick-tables of 1 million entries size. > > Is peers protocol single threaded and high usage on single core is expected > or am I experiencing some kind of bug? I'll keep digging to be sure nothing > else from my configuration is causing the issue. Almost. More precisely, each peers connection runs on a single thread at once (like any connection). Some such connections may experience heavy protocol parsing so it may be possible that sometimes you end up with an unbalance number of connections on threads. It's tricky though, because we could imagine a mechanism to try to evenly spread the outgoing peers connection on threads but the incoming ones will land where they land. That's something you can check with "show sess" and/or "show fd", looking for those related to your peers and checking their thread_mask. If you find two on the same thread (same thread_mask), you can shut one of them down using "shutdown session " and it will reconnect, possibly on another thread. That could confirm that it's the root cause of the problem you're experiencing. I'm wondering if we shouldn't introduce a notion of "heavy connection" like we already have heavy tasks for the scheduler. These ones would be balanced differently from others. Usually they're long-lived and can eat massive amounts of CPU so it would make sense. The only ones I'm thinking about are the peers ones but the concept would be more portable than focusing on peers. Usually such long connections are not refreshed often so we probably prefer to spend time carefully picking the best thread rather than saving 200ns of processing and having to pay it for the whole connection's life time. Willy
Re: Always add "reason"
Hi Marco,
On Thu, Mar 03, 2022 at 12:26:09PM +0100, Marco Corte wrote:
> Hi!
>
> I can add a "reason phrase" to a response based on the HTTP status like
> this:
>
> http-response set-status 200 reason OK if { status eq 200 }
>
> Is there any way to add the reason phrase for a set of codes without an
> explicit rule for each code?
> I would like to write a set of rules like this
>
> http-response set-status 200 reason OK if { status eq 200 }
> http-response set-status %[status] reason NotOK unless { status eq 200 }
Unfortunately I don't see a way to easily do this. I didn't even
remembered that the internal representation for the reason was
conveyed, I thought it was built when serializing the status on the
wire.
In fact, since the advent of HTTP/2 which completely dropped the reason
phrase, you really cannot trust that element anymore at all. It used to
be very unreliable even before HTTP/2 since plenty of proxies were
placing their own reason for a given code, but nowadays it's really
impossible to trust it anymore.
Out of curiosity, what would be your use case ?
Thanks,
Willy
