Re: Peers using heavily single cpu core
Hi Maciej, On Mon, May 23, 2022 at 08:50:53AM +0200, Maciej Zdeb wrote: > Hi Christopher, > I've verified that outgoing connections are now spread between multiple > threads! Thank you very much! That's really great, thank you for testing! I, too, thought it was worth being merged even this late in the dev cycle. The way it was done allows to postpone conversion of other applets, so the risk of breaking something remains fairly limited. We'll see. Cheers, Willy
Re: Peers using heavily single cpu core
Hi Christopher, I've verified that outgoing connections are now spread between multiple threads! Thank you very much! 23 : st=0x000121(cl heopI W:sRa R:srA) tmask=0x8 umask=0x0 owner=0x56219da29280 iocb=0x56219cdc8730(sock_conn_iocb) back=0 cflg=0x0300 fam=ipv4 lport=1024 rport=52046 fe=hap1 mux=PASS ctx=0x7f4c2002f410 xprt=RAW 24 : st=0x000121(cl heopI W:sRa R:srA) tmask=0x20 umask=0x0 owner=0x7f4c2802a7a0 iocb=0x56219cdc8730(sock_conn_iocb) back=0 cflg=0x0300 fam=ipv4 lport=1024 rport=55428 fe=hap1 mux=PASS ctx=0x7f4c2c02f200 xprt=RAW 25 : st=0x000121(cl heopI W:sRa R:srA) tmask=0x1 umask=0x0 owner=0x7f4c2c0260b0 iocb=0x56219cdc8730(sock_conn_iocb) back=0 cflg=0x0300 fam=ipv4 lport=1024 rport=51190 fe=hap1 mux=PASS ctx=0x56219da24180 xprt=RAW 26 : st=0x010121(cL heopI W:sRa R:srA) tmask=0x20 umask=0x0 owner=0x7f4c2c026ac0 iocb=0x56219cdc8730(sock_conn_iocb) back=1 cflg=0x1300 fam=ipv4 lport=34454 rport=1024 px=hap1 mux=PASS ctx=0x7f4c2c026610 xprt=RAW 27 : st=0x000121(cl heopI W:sRa R:srA) tmask=0x4 umask=0x0 owner=0x7f4c40026cf0 iocb=0x56219cdc8730(sock_conn_iocb) back=0 cflg=0x0300 fam=ipv4 lport=1024 rport=50226 fe=hap1 mux=PASS ctx=0x7f4c3002eb60 xprt=RAW Kind regards, wt., 17 maj 2022 o 16:25 Christopher Faulet napisał(a): > Le 4/20/22 à 14:51, Maciej Zdeb a écrit : > > Hi Willy, > > I saw Christopher changes are now merged. I was wondering how to proceed > with my > > issue. Right now in stream_new() I'm able to get cs_endpoint and appctx > (if > > endpoint is applet), so I can get thread_mask of appctx to create a > stream task > > on the same thread. Is this approach correct? > > > > Hi Maciej, > > I've finally finish my applet refactoring. Now it is possible to choose > where to > start an applet. I've also merged your patches. Peer applets must now be > balanced across threads. So, you may give it a try to be sure it solves > your > issue and also validate everything works fine. > > -- > Christopher Faulet >
Re: Peers using heavily single cpu core
Le 4/20/22 à 14:51, Maciej Zdeb a écrit : Hi Willy, I saw Christopher changes are now merged. I was wondering how to proceed with my issue. Right now in stream_new() I'm able to get cs_endpoint and appctx (if endpoint is applet), so I can get thread_mask of appctx to create a stream task on the same thread. Is this approach correct? Hi Maciej, I've finally finish my applet refactoring. Now it is possible to choose where to start an applet. I've also merged your patches. Peer applets must now be balanced across threads. So, you may give it a try to be sure it solves your issue and also validate everything works fine. -- Christopher Faulet
Re: Peers using heavily single cpu core
No worries, thanks for the update. Kind regards, Maciej śr., 20 kwi 2022 o 21:59 Willy Tarreau napisał(a): > Hi Maciej, > > On Wed, Apr 20, 2022 at 02:51:32PM +0200, Maciej Zdeb wrote: > > Hi Willy, > > I saw Christopher changes are now merged. I was wondering how to proceed > > with my issue. Right now in stream_new() I'm able to get cs_endpoint and > > appctx (if endpoint is applet), so I can get thread_mask of appctx to > > create a stream task on the same thread. Is this approach correct? > > It's still not possible at this stage. The related modifications still > have to be done. We've discussed about this today and these still seem > possible but it's difficult to estimate if it will be done easily or > if we'll discover new showstoppers. I'd suggest that you wait a bit > more, I'm afraid. > > Cheers, > Willy >
Re: Peers using heavily single cpu core
Hi Maciej, On Wed, Apr 20, 2022 at 02:51:32PM +0200, Maciej Zdeb wrote: > Hi Willy, > I saw Christopher changes are now merged. I was wondering how to proceed > with my issue. Right now in stream_new() I'm able to get cs_endpoint and > appctx (if endpoint is applet), so I can get thread_mask of appctx to > create a stream task on the same thread. Is this approach correct? It's still not possible at this stage. The related modifications still have to be done. We've discussed about this today and these still seem possible but it's difficult to estimate if it will be done easily or if we'll discover new showstoppers. I'd suggest that you wait a bit more, I'm afraid. Cheers, Willy
Re: Peers using heavily single cpu core
Hi Willy, I saw Christopher changes are now merged. I was wondering how to proceed with my issue. Right now in stream_new() I'm able to get cs_endpoint and appctx (if endpoint is applet), so I can get thread_mask of appctx to create a stream task on the same thread. Is this approach correct? Kind regards czw., 7 kwi 2022 o 22:32 Maciej Zdeb napisał(a): > Hi Willy, > I adjusted the 3rd patch to use HA_ATOMIC_INC/HA_ATOMIC_DEC. Thanks! > > czw., 7 kwi 2022 o 14:02 Willy Tarreau napisał(a): > >> There are some ongoing changes in that area that are about to be merged >> (probably early >> next week) and that will help for this task, so we'll have a closer look, >> and it's very likely that we could proceed like this soon. >> > > Great! I must admit that I imagined a stream_new_on() function but after > looking at stream_new() I thought I'd better ask for help. ;-) > > Kind regards, > Maciej > >
Re: Peers using heavily single cpu core
Hi Willy, I adjusted the 3rd patch to use HA_ATOMIC_INC/HA_ATOMIC_DEC. Thanks! czw., 7 kwi 2022 o 14:02 Willy Tarreau napisał(a): > There are some ongoing changes in that area that are about to be merged > (probably early > next week) and that will help for this task, so we'll have a closer look, > and it's very likely that we could proceed like this soon. > Great! I must admit that I imagined a stream_new_on() function but after looking at stream_new() I thought I'd better ask for help. ;-) Kind regards, Maciej 0002-MINOR-peers-Track-number-of-applets-run-by-thread.patch Description: Binary data 0003-MINOR-peers-Balance-applets-across-threads.patch Description: Binary data 0001-MINOR-applet-Add-function-to-create-applet-on-select.patch Description: Binary data
Re: Peers using heavily single cpu core
On Thu, Apr 07, 2022 at 11:50:33AM +0200, Willy Tarreau wrote: > And in an ideal world, the applet's init code could be the one responsible > for creating its stream. Historically it wasn't even possible because both > contexts were strongly tied. But with the changes Christopher recently did > to keep the applets away, maybe that's something we can think of. If so, > that would also make the Lua client and http_client setup code so much > cleaner, I'll discuss this with Christopher to see what's missing to > reach that point, and will get back to you ASAP (or please yell at me > if you don't hear about me for some time). So after discussing about this, it seems to make sense. There are some ongoing changes in that area that are about to be merged (probably early next week) and that will help for this task, so we'll have a closer look, and it's very likely that we could proceed like this soon. Stay tuned! Willy
Re: Peers using heavily single cpu core
Hi Maciej, On Wed, Apr 06, 2022 at 11:17:44AM +0200, Maciej Zdeb wrote: > Hi Willy, > > I've managed to implement the easy part (attached patches). Thanks for this! I'm having a minor comment on the 3rd patch, see at the end. > However this is > not enough to get it to work properly. I'm stuck in a debug of following > stack trace (BUG_ON task_queue() function): > > FATAL: bug condition "task->thread_mask != tid_bit" matched at > include/haproxy/task.h:320 > call trace(12): > | 0x55f921169be3 [c7 04 25 01 00 00 00 00]: main+0x112f53 > | 0x55f92116c4ee [48 8b 43 08 48 8b 78 10]: si_applet_wake_cb+0x7e/0xf3 > | 0x55f9211cc276 [48 8b 43 08 48 8b 40 10]: task_run_applet+0x196/0x5bd > | 0x55f92118e6ca [48 89 c5 eb 0f 90 4c 89]: > run_tasks_from_lists+0x3aa/0x871 > | 0x55f92118ef73 [29 44 24 14 8b 7c 24 14]: > process_runnable_tasks+0x3d3/0x892 > | 0x55f92115eb44 [83 3d f9 ef 1d 00 01 0f]: run_poll_loop+0x124/0x3d4 > | 0x55f92115efc9 [48 8b 1d d0 94 12 00 4c]: main+0x108339 > | 0x55f9210589ff [31 c0 e8 ba a6 16 00 31]: main+0x1d6f/0x296a > > I'm not sure why it happens. I suspect that it happens because of > session_new() in peer_session_create() that creates another task on the > current thread. If a session task tries to wake up a peer applet task then > it would explain the stack trace, however I've failed to fully understand > the "flow" between applet and session (and how/when task_queue is called in > this flow). I would appreciate any help or tips on how to proceed. I hadn't thought about that, as usual the devil is in the details :-( The problem here is that it's not allowed to requeue a task that belongs to another thread, and that's exactly what is being done there, because while the applet runs on its target thread, it's bound to a stream that was programmed for another thread, and when requeuing its task it fails. We could imagine having a stream_new_on() but that's not realistic because stream_new() already touches some thread-specific stuff and even calls the frontend's accept() function or initializes streams. Thus if we sum up, it's not possible to create a stream for another thread, but functionally speaking, that's not something we absolutely want to do, we would also be fine with having someone else create such a stream. And in an ideal world, the applet's init code could be the one responsible for creating its stream. Historically it wasn't even possible because both contexts were strongly tied. But with the changes Christopher recently did to keep the applets away, maybe that's something we can think of. If so, that would also make the Lua client and http_client setup code so much cleaner, I'll discuss this with Christopher to see what's missing to reach that point, and will get back to you ASAP (or please yell at me if you don't hear about me for some time). Regarding the comment on the 3rd patch: > From c7f1be0dab823b81a1806d8eaca906d9c2483ac8 Mon Sep 17 00:00:00 2001 > From: Maciej Zdeb > Date: Mon, 4 Apr 2022 13:08:11 +0200 > Subject: [PATCH 3/3] MINOR: peers: Balance applets across threads > (...) > @@ -3236,7 +3240,7 @@ static struct appctx *peer_session_create(struct peers > *peers, struct peer *peer > > s->res.flags |= CF_READ_DONTWAIT; > > - peers->applet_count[tid]++; > + peers->applet_count[thr]++; You'll need to switch to: HA_ATOMIC_INC(&peers->applet_count[thr]); instead. Indeed, since the counter belongs to another thread, any other thread could be manipulating it at the same time. thanks! Willy
Re: Peers using heavily single cpu core
Hi Willy,
I've managed to implement the easy part (attached patches). However this is
not enough to get it to work properly. I'm stuck in a debug of following
stack trace (BUG_ON task_queue() function):
FATAL: bug condition "task->thread_mask != tid_bit" matched at
include/haproxy/task.h:320
call trace(12):
| 0x55f921169be3 [c7 04 25 01 00 00 00 00]: main+0x112f53
| 0x55f92116c4ee [48 8b 43 08 48 8b 78 10]: si_applet_wake_cb+0x7e/0xf3
| 0x55f9211cc276 [48 8b 43 08 48 8b 40 10]: task_run_applet+0x196/0x5bd
| 0x55f92118e6ca [48 89 c5 eb 0f 90 4c 89]:
run_tasks_from_lists+0x3aa/0x871
| 0x55f92118ef73 [29 44 24 14 8b 7c 24 14]:
process_runnable_tasks+0x3d3/0x892
| 0x55f92115eb44 [83 3d f9 ef 1d 00 01 0f]: run_poll_loop+0x124/0x3d4
| 0x55f92115efc9 [48 8b 1d d0 94 12 00 4c]: main+0x108339
| 0x55f9210589ff [31 c0 e8 ba a6 16 00 31]: main+0x1d6f/0x296a
I'm not sure why it happens. I suspect that it happens because of
session_new() in peer_session_create() that creates another task on the
current thread. If a session task tries to wake up a peer applet task then
it would explain the stack trace, however I've failed to fully understand
the "flow" between applet and session (and how/when task_queue is called in
this flow). I would appreciate any help or tips on how to proceed.
Kind regards,
pt., 1 kwi 2022 o 14:34 Willy Tarreau napisał(a):
> On Fri, Apr 01, 2022 at 11:23:34AM +0200, Maciej Zdeb wrote:
> > > I remember mentioning something about trying to change this in the
> future.
> > > For outgoing connections I think we could try to figure where peers
> > > connections are and always use one least represented thread.
> > >
> > If I understand the code correctly, the peer_session_create() function
> > creates appctx by appctx_new() which allocates all outgoing connections
> on
> > the current thread "appctx->t = task_new_here();"?
> > So changing peer_session_create() into:
> >
> > static struct appctx *peer_session_create(struct peers *peers, struct
> peer
> > *peer, int thread) {
> > ...
> > appctx = appctx_new_on(&peer_applet, thread);
> > if (!appctx)
> > goto out_close;
> > ...
> > }
> >
> > where appctx_new_on() is a new function that creates an applet on a
> > specified thread would fix my issue? I'm worrying it's not that easy and
> > some locking here and there might be needed. ;-)
>
> Indeed, I think it would work without problems! In this case you could
> even add an array of MAXTHREADS to the peers struct to count the number
> of applets per thread, then peer_session_create() could just scan the
> threads to find the least loaded one and use this one. This would easily
> do the job and sounds simple and efficient enough to get into 2.6.
>
> Please, if you go that way, think about splitting your work in at least
> 3 patches:
> - 1 for appctx_new_on()
> - 1 to maintain the applet count per peers struct
> - 1 to use that counter to start the applet on the best thread
>
> You may find that you need to implement more stuff, but you get the idea.
>
> thanks,
> Willy
>
0001-MINOR-applet-Add-function-to-create-applet-on-select.patch
Description: Binary data
0002-MINOR-peers-Track-number-of-applets-run-by-thread.patch
Description: Binary data
0003-MINOR-peers-Balance-applets-across-threads.patch
Description: Binary data
Re: Peers using heavily single cpu core
On Fri, Apr 01, 2022 at 11:23:34AM +0200, Maciej Zdeb wrote:
> > I remember mentioning something about trying to change this in the future.
> > For outgoing connections I think we could try to figure where peers
> > connections are and always use one least represented thread.
> >
> If I understand the code correctly, the peer_session_create() function
> creates appctx by appctx_new() which allocates all outgoing connections on
> the current thread "appctx->t = task_new_here();"?
> So changing peer_session_create() into:
>
> static struct appctx *peer_session_create(struct peers *peers, struct peer
> *peer, int thread) {
> ...
> appctx = appctx_new_on(&peer_applet, thread);
> if (!appctx)
> goto out_close;
> ...
> }
>
> where appctx_new_on() is a new function that creates an applet on a
> specified thread would fix my issue? I'm worrying it's not that easy and
> some locking here and there might be needed. ;-)
Indeed, I think it would work without problems! In this case you could
even add an array of MAXTHREADS to the peers struct to count the number
of applets per thread, then peer_session_create() could just scan the
threads to find the least loaded one and use this one. This would easily
do the job and sounds simple and efficient enough to get into 2.6.
Please, if you go that way, think about splitting your work in at least
3 patches:
- 1 for appctx_new_on()
- 1 to maintain the applet count per peers struct
- 1 to use that counter to start the applet on the best thread
You may find that you need to implement more stuff, but you get the idea.
thanks,
Willy
Re: Peers using heavily single cpu core
pt., 1 kwi 2022 o 09:06 Willy Tarreau napisał(a):
> I seem to remember a discussion about this in the past and that the
> conclusion basically was that outgoing connections are sent on the
> "current" thread when the management task creates them, while incoming
> connections are balanced depending on the overall process' load.
Yes, that's what I'm observing. High cpu usage is caused by outgoing
connections (visible on "show fd" dump - rport=1024) and currently I don't
mind the incoming ones because they are balanced pretty well.
> I remember mentioning something about trying to change this in the future.
> For outgoing connections I think we could try to figure where peers
> connections are and always use one least represented thread.
>
If I understand the code correctly, the peer_session_create() function
creates appctx by appctx_new() which allocates all outgoing connections on
the current thread "appctx->t = task_new_here();"?
So changing peer_session_create() into:
static struct appctx *peer_session_create(struct peers *peers, struct peer
*peer, int thread) {
...
appctx = appctx_new_on(&peer_applet, thread);
if (!appctx)
goto out_close;
...
}
where appctx_new_on() is a new function that creates an applet on a
specified thread would fix my issue? I'm worrying it's not that easy and
some locking here and there might be needed. ;-)
Kind regards,
Maciej
> Willy
>
Re: Peers using heavily single cpu core
On Fri, Apr 01, 2022 at 08:58:43AM +0200, Maciej Zdeb wrote: > Hi Willy, > Thanks for the patch! Now I'll be able to rebalance peers connections > manually. Do you know why so many of them are landing on the same thread? I seem to remember a discussion about this in the past and that the conclusion basically was that outgoing connections are sent on the "current" thread when the management task creates them, while incoming connections are balanced depending on the overall process' load. I remember mentioning something about trying to change this in the future. For outgoing connections I think we could try to figure where peers connections are and always use one least represented thread. For incoming ones it could be a bit trickier because it would require to implement an incoming load-balancing mechanism that is specific to the listener. That wouldn't be stupid in the end, given that we already had to do something specific for QUIC (disable inbound balancing). The difficulty with the peers is that an incoming connection may replace an existing one and we can't know which one. But given that peers connections do not need to deal with high rates and that there are often more threads than connections, we could very well use a max-accept 1 on these ones and carefully scan all threads for peers connections to spot the best one. Willy
Re: Peers using heavily single cpu core
Hi Willy, Thanks for the patch! Now I'll be able to rebalance peers connections manually. Do you know why so many of them are landing on the same thread? Kind regards czw., 31 mar 2022 o 15:16 Willy Tarreau napisał(a): > On Thu, Mar 31, 2022 at 11:56:13AM +0200, Willy Tarreau wrote: > > > I've made some "session shutdown" tests. With simple configuration > > > (attached) it is very easy to reproduce a situation when "shutdown > session" > > > works only for the first time. > > > > Indeed, thanks for this, I can as well reproduce it. With a bit of luck > > I'll figure what's happening, hoping I won't unearth an ugly bug! > > Now found and fixed. It's been bogus since we've split the list of streams > to be per-thread in 2.4, patch attached if you need it. > > Thanks, > Willy >
Re: Peers using heavily single cpu core
On Thu, Mar 31, 2022 at 11:56:13AM +0200, Willy Tarreau wrote:
> > I've made some "session shutdown" tests. With simple configuration
> > (attached) it is very easy to reproduce a situation when "shutdown session"
> > works only for the first time.
>
> Indeed, thanks for this, I can as well reproduce it. With a bit of luck
> I'll figure what's happening, hoping I won't unearth an ugly bug!
Now found and fixed. It's been bogus since we've split the list of streams
to be per-thread in 2.4, patch attached if you need it.
Thanks,
Willy
>From c40c407268190e41d7d0238e3e2b7669147df9f7 Mon Sep 17 00:00:00 2001
From: Willy Tarreau
Date: Thu, 31 Mar 2022 14:49:45 +0200
Subject: BUG/MINOR: cli/stream: fix "shutdown session" to iterate over all
threads
The list of streams was modified in 2.4 to become per-thread with commit
a698eb673 ("MINOR: streams: use one list per stream instead of a global
one"). However the change applied to cli_parse_shutdown_session() is
wrong, as it uses the nullity of the stream pointer to continue on next
threads, but this one is not null once the list_for_each_entry() loop
is finished, it points to the list's head again, so the loop doesn't
check other threads, and no message is printed either to say that the
stream was not found.
Instead we should check if the stream is equal to the requested pointer
since this is the condition to break out of the loop.
Thus must be backported to 2.4. Thanks to Maciej Zdeb for reporting this.
---
src/stream.c | 10 +-
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/src/stream.c b/src/stream.c
index 8acce8075..fd8dada12 100644
--- a/src/stream.c
+++ b/src/stream.c
@@ -3750,16 +3750,16 @@ static int cli_parse_shutdown_session(char **args, char
*payload, struct appctx
if (!cli_has_level(appctx, ACCESS_LVL_ADMIN))
return 1;
- if (!*args[2])
- return cli_err(appctx, "Session pointer expected (use 'show
sess').\n");
-
ptr = (void *)strtoul(args[2], NULL, 0);
+ if (!ptr)
+ return cli_err(appctx, "Session pointer expected (use 'show
sess').\n");
+
strm = NULL;
thread_isolate();
/* first, look for the requested stream in the stream table */
- for (thr = 0; !strm && thr < global.nbthread; thr++) {
+ for (thr = 0; strm != ptr && thr < global.nbthread; thr++) {
list_for_each_entry(strm, &ha_thread_ctx[thr].streams, list) {
if (strm == ptr) {
stream_shutdown(strm, SF_ERR_KILLED);
@@ -3771,7 +3771,7 @@ static int cli_parse_shutdown_session(char **args, char
*payload, struct appctx
thread_release();
/* do we have the stream ? */
- if (!strm)
+ if (strm != ptr)
return cli_err(appctx, "No such session (use 'show sess').\n");
return 1;
--
2.35.1
Re: Peers using heavily single cpu core
Hi Maciej, On Wed, Mar 16, 2022 at 12:21:59PM +0100, Maciej Zdeb wrote: > Hi Willy, > did you find anything? I forgot about it after switching to plenty of other topics since :-( > I've made some "session shutdown" tests. With simple configuration > (attached) it is very easy to reproduce a situation when "shutdown session" > works only for the first time. Indeed, thanks for this, I can as well reproduce it. With a bit of luck I'll figure what's happening, hoping I won't unearth an ugly bug! Willy
Re: Peers using heavily single cpu core
Hi Willy, did you find anything? I've made some "session shutdown" tests. With simple configuration (attached) it is very easy to reproduce a situation when "shutdown session" works only for the first time. 1. Kill haproxy to have a clean start: root@hap-rnd-1a:/etc/haproxy# killall haproxy 2. Show current sessions: root@hap-rnd-1a:/etc/haproxy# echo "show sess" | socat unix-connect:/var/run/haproxy/haproxy1.sock stdio 0x562291719560: proto=tcpv4 src=10.0.0.3:5276 fe=hap-rnd-1a be=hap-rnd-1a srv= ts=00 epoch=0 age=4s calls=1 rate=0 cpu=0 lat=0 rq[f=c48202h,i=0,an=00h,rx=,wx=,ax=] rp[f=80048202h,i=0,an=00h,rx=,wx=,ax=] s0=[8,200048h,fd=29,ex=] s1=[8,204058h,fd=-1,ex=] exp= 0x562291719fa0: proto=unix_stream src=unix:1 fe=GLOBAL be= srv= ts=00 epoch=0 age=0s calls=2 rate=2 cpu=0 lat=0 rq[f=c0c020h,i=0,an=00h,rx=,wx=,ax=] rp[f=80008002h,i=0,an=00h,rx=,wx=,ax=] s0=[8,280008h,fd=30,ex=] s1=[8,204018h,fd=-1,ex=] exp=30s 3. Kill peer session 0x562291719560 root@hap-rnd-1a:/etc/haproxy# echo "shutdown session 0x562291719560" | socat unix-connect:/var/run/haproxy/haproxy1.sock stdio 4. Show current sessions - confirm peer connection shutdown root@hap-rnd-1a:/etc/haproxy# echo "show sess" | socat unix-connect:/var/run/haproxy/haproxy1.sock stdio 0x7f2628026220: proto=unix_stream src=unix:1 fe=GLOBAL be= srv= ts=00 epoch=0x1 age=0s calls=1 rate=1 cpu=0 lat=0 rq[f=c08000h,i=0,an=00h,rx=,wx=,ax=] rp[f=80008000h,i=0,an=00h,rx=,wx=,ax=] s0=[8,240008h,fd=29,ex=] s1=[8,204018h,fd=-1,ex=] exp= 5. Show current sessions - confirm new peer connection (note that new connection has same id 0x7f2628026220 as unix_stream in previous output, but it is probably by chance) root@hap-rnd-1a:/etc/haproxy# echo "show sess" | socat unix-connect:/var/run/haproxy/haproxy1.sock stdio 0x7f2628026220: proto=tcpv4 src=10.0.0.3:5288 fe=hap-rnd-1a be=hap-rnd-1a srv= ts=00 epoch=0x2 age=3s calls=1 rate=0 cpu=0 lat=0 rq[f=c48200h,i=0,an=00h,rx=,wx=,ax=] rp[f=80048202h,i=0,an=00h,rx=,wx=,ax=] s0=[8,200048h,fd=29,ex=] s1=[8,204058h,fd=-1,ex=] exp= 0x7f263c026220: proto=unix_stream src=unix:1 fe=GLOBAL be= srv= ts=00 epoch=0x2 age=0s calls=1 rate=1 cpu=0 lat=0 rq[f=c08000h,i=0,an=00h,rx=,wx=,ax=] rp[f=80008002h,i=0,an=00h,rx=,wx=,ax=] s0=[8,240008h,fd=30,ex=] s1=[8,204018h,fd=-1,ex=] exp= 6. Again kill peer session 0x7f2628026220 root@hap-rnd-1a:/etc/haproxy# echo "shutdown session 0x7f2628026220" | socat unix-connect:/var/run/haproxy/haproxy1.sock stdio 7. Show current sessions - note that 0x7f2628026220 was not killed in the previous step root@hap-rnd-1a:/etc/haproxy# echo "show sess" | socat unix-connect:/var/run/haproxy/haproxy1.sock stdio 0x7f267c026a60: proto=unix_stream src=unix:1 fe=GLOBAL be= srv= ts=00 epoch=0x3 age=0s calls=1 rate=1 cpu=0 lat=0 rq[f=c08000h,i=0,an=00h,rx=,wx=,ax=] rp[f=80008000h,i=0,an=00h,rx=,wx=,ax=] s0=[8,240008h,fd=30,ex=] s1=[8,204018h,fd=-1,ex=] exp= 0x7f2628026220: proto=tcpv4 src=10.0.0.3:5288 fe=hap-rnd-1a be=hap-rnd-1a srv= ts=00 epoch=0x2 age=17s calls=1 rate=0 cpu=0 lat=0 rq[f=c48202h,i=0,an=00h,rx=,wx=,ax=] rp[f=80048202h,i=0,an=00h,rx=,wx=,ax=] s0=[8,200048h,fd=29,ex=] s1=[8,204058h,fd=-1,ex=] exp= 8. Kill peer session 0x7f2628026220 root@hap-rnd-1a:/etc/haproxy# echo "shutdown session 0x7f2628026220" | socat unix-connect:/var/run/haproxy/haproxy1.sock stdio 9. Show session - again no effect root@hap-rnd-1a:/etc/haproxy# echo "show sess" | socat unix-connect:/var/run/haproxy/haproxy1.sock stdio 0x7f2628026220: proto=tcpv4 src=10.0.0.3:5288 fe=hap-rnd-1a be=hap-rnd-1a srv= ts=00 epoch=0x2 age=22s calls=1 rate=0 cpu=0 lat=0 rq[f=c48202h,i=0,an=00h,rx=,wx=,ax=] rp[f=80048202h,i=0,an=00h,rx=,wx=,ax=] s0=[8,200048h,fd=29,ex=] s1=[8,204058h,fd=-1,ex=] exp= 0x7f261c026220: proto=unix_stream src=unix:1 fe=GLOBAL be= srv= ts=00 epoch=0x4 age=0s calls=1 rate=1 cpu=0 lat=0 rq[f=c08000h,i=0,an=00h,rx=,wx=,ax=] rp[f=80008002h,i=0,an=00h,rx=,wx=,ax=] s0=[8,240008h,fd=30,ex=] s1=[8,204018h,fd=-1,ex=] exp= Kind regards, sob., 12 mar 2022 o 00:20 Willy Tarreau napisał(a): > On Fri, Mar 11, 2022 at 10:19:09PM +0100, Maciej Zdeb wrote: > > Hi Willy, > > > > Thank you for such useful info! I've checked the worst HAProxy nodes and > on > > every such node all outgoing peers connections are run on the same > thread: > (...) > Indeed. I was pretty sure we were setting them to any thread on creation > but maybe I'm wrong, I'll have to recheck. > > > On one node I was able to rebalance it, but on the node above (and other > > nodes) I'm not able to shutdown the sessions: > (...) > > echo "shutdown session 0x7f0aa402e2c0" | socat > > unix-connect:/var/run/haproxy.sock stdio > > > > echo "show sess 0x7f0aa402e2c0" | socat > unix-connect:/var/run/haproxy.sock > > stdio > > 0x7f0aa402e2c0: [11/Mar/2022:07:17:02.313221] id=0 proto=? > (...) > > That's not expected, another thing I'll have to check. > > Thanks for testing. I'll put that in pause for the week-end, though :-) > > cheers, > Willy > global chroot /var/lib/h
Re: Peers using heavily single cpu core
On Fri, Mar 11, 2022 at 10:19:09PM +0100, Maciej Zdeb wrote: > Hi Willy, > > Thank you for such useful info! I've checked the worst HAProxy nodes and on > every such node all outgoing peers connections are run on the same thread: (...) Indeed. I was pretty sure we were setting them to any thread on creation but maybe I'm wrong, I'll have to recheck. > On one node I was able to rebalance it, but on the node above (and other > nodes) I'm not able to shutdown the sessions: (...) > echo "shutdown session 0x7f0aa402e2c0" | socat > unix-connect:/var/run/haproxy.sock stdio > > echo "show sess 0x7f0aa402e2c0" | socat unix-connect:/var/run/haproxy.sock > stdio > 0x7f0aa402e2c0: [11/Mar/2022:07:17:02.313221] id=0 proto=? (...) That's not expected, another thing I'll have to check. Thanks for testing. I'll put that in pause for the week-end, though :-) cheers, Willy
Re: Peers using heavily single cpu core
Hi Willy, Thank you for such useful info! I've checked the worst HAProxy nodes and on every such node all outgoing peers connections are run on the same thread: echo "show fd" | socat unix-connect:/var/run/haproxy.sock stdio | grep "px=" 430 : st=0x010121(cL heopI W:sRa R:srA) tmask=0x40 umask=0x0 owner=0x7f0aa4039250 iocb=0x564c0df95f40(sock_conn_iocb) back=1 cflg=0x10002300 fam=ipv4 lport=22850 rport=1024 px=hap. mux=PASS ctx=0x7f0aa4039450 xprt=RAW 500 : st=0x010121(cL heopI W:sRa R:srA) tmask=0x40 umask=0x0 owner=0x7f0aa403d4e0 iocb=0x564c0df95f40(sock_conn_iocb) back=1 cflg=0x10002300 fam=ipv4 lport=58906 rport=1024 px=hap mux=PASS ctx=0x7f0aa403d6e0 xprt=RAW 501 : st=0x010121(cL heopI W:sRa R:srA) tmask=0x40 umask=0x0 owner=0x7f0aa4041770 iocb=0x564c0df95f40(sock_conn_iocb) back=1 cflg=0x10002300 fam=ipv4 lport=54470 rport=1024 px=hap mux=PASS ctx=0x7f0aa4041970 xprt=RAW 502 : st=0x010121(cL heopI W:sRa R:srA) tmask=0x40 umask=0x0 owner=0x7f0aa4045a00 iocb=0x564c0df95f40(sock_conn_iocb) back=1 cflg=0x10002300 fam=ipv4 lport=48926 rport=1024 px=hap mux=PASS ctx=0x7f0aa4045c00 xprt=RAW 503 : st=0x010121(cL heopI W:sRa R:srA) tmask=0x40 umask=0x0 owner=0x7f0aa4049c90 iocb=0x564c0df95f40(sock_conn_iocb) back=1 cflg=0x10002300 fam=ipv4 lport=21468 rport=1024 px=hap mux=PASS ctx=0x7f0aa4049e90 xprt=RAW echo "show sess" | socat unix-connect:/var/run/haproxy.sock stdio | grep "proto=?" | grep "epoch=0 " | grep 14h 0x7f0aa402e2c0: proto=? ts=00 epoch=0 age=14h53m calls=10724 rate=1 cpu=0 lat=0 rq[f=848202h,i=0,an=00h,rx=5s,wx=,ax=] rp[f=80448202h,i=0,an=00h,rx=,wx=,ax=] s0=[8,204048h,fd=-1,ex=] s1=[8,2000d8h,fd=430,ex=] exp=3s 0x7f0aa402ece0: proto=? ts=00 epoch=0 age=14h53m calls=10724 rate=1 cpu=0 lat=0 rq[f=848202h,i=0,an=00h,rx=5s,wx=,ax=] rp[f=80448202h,i=0,an=00h,rx=,wx=,ax=] s0=[8,204048h,fd=-1,ex=] s1=[8,2000d8h,fd=500,ex=] exp=3s 0x7f0aa402f700: proto=? ts=00 epoch=0 age=14h53m calls=10724 rate=1 cpu=0 lat=0 rq[f=848202h,i=0,an=00h,rx=5s,wx=,ax=] rp[f=80448202h,i=0,an=00h,rx=,wx=,ax=] s0=[8,204048h,fd=-1,ex=] s1=[8,2000d8h,fd=501,ex=] exp=3s 0x7f0aa4030120: proto=? ts=00 epoch=0 age=14h53m calls=10722 rate=0 cpu=0 lat=0 rq[f=848202h,i=0,an=00h,rx=5s,wx=,ax=] rp[f=80448202h,i=0,an=00h,rx=,wx=,ax=] s0=[8,204048h,fd=-1,ex=] s1=[8,2000d8h,fd=502,ex=] exp=2s 0x7f0aa4030b40: proto=? ts=00 epoch=0 age=14h53m calls=10724 rate=1 cpu=0 lat=0 rq[f=848202h,i=0,an=00h,rx=5s,wx=,ax=] rp[f=80448202h,i=0,an=00h,rx=,wx=,ax=] s0=[8,204048h,fd=-1,ex=] s1=[8,2000d8h,fd=503,ex=] exp=3s On one node I was able to rebalance it, but on the node above (and other nodes) I'm not able to shutdown the sessions: echo "show sess 0x7f0aa402e2c0" | socat unix-connect:/var/run/haproxy.sock stdio 0x7f0aa402e2c0: [11/Mar/2022:07:17:02.313221] id=0 proto=? flags=0x6, conn_retries=3, srv_conn=(nil), pend_pos=(nil) waiting=0 epoch=0 frontend=x (id=4294967295 mode=http), listener=? (id=0) backend=x (id=4294967295 mode=http) addr=x:22850 server= (id=0) addr=y:1024 task=0x7f0aa402e7d0 (state=0x00 nice=0 calls=10741 rate=0 exp=1s tmask=0x40 age=14h54m) si[0]=0x7f0aa402e600 (state=EST flags=0x204048 endp0=APPCTX:0x7f0aa402df90 exp= et=0x000 sub=0) si[1]=0x7f0aa402e658 (state=EST flags=0x2000d8 endp1=CS:0x7f0aa4039400 exp= et=0x000 sub=1) app0=0x7f0aa402df90 st0=7 st1=0 st2=0 applet= tmask=0x40 nice=0 calls=206051832 rate=2860 cpu=0 lat=0 co1=0x7f0aa4039250 ctrl=tcpv4 xprt=RAW mux=PASS data=STRM target=PROXY:0x564c0ff53760 flags=0x10003300 fd=430 fd.state=10121 updt=0 fd.tmask=0x40 cs=0x7f0aa4039400 csf=0x8200 ctx=(nil) req=0x7f0aa402e2d0 (f=0x848202 an=0x0 pipe=0 tofwd=-1 total=9489764648) an_exp= rex=4s wex= buf=0x7f0aa402e2d8 data=(nil) o=0 p=0 i=0 size=0 res=0x7f0aa402e330 (f=0x80448202 an=0x0 pipe=0 tofwd=-1 total=9320288685) an_exp= rex= wex= buf=0x7f0aa402e338 data=(nil) o=0 p=0 i=0 size=0 echo "shutdown session 0x7f0aa402e2c0" | socat unix-connect:/var/run/haproxy.sock stdio echo "show sess 0x7f0aa402e2c0" | socat unix-connect:/var/run/haproxy.sock stdio 0x7f0aa402e2c0: [11/Mar/2022:07:17:02.313221] id=0 proto=? flags=0x6, conn_retries=3, srv_conn=(nil), pend_pos=(nil) waiting=0 epoch=0 frontend=x (id=4294967295 mode=http), listener=? (id=0) backend=x (id=4294967295 mode=http) addr=x:22850 server= (id=0) addr=x:1024 task=0x7f0aa402e7d0 (state=0x00 nice=0 calls=10745 rate=1 exp=4s tmask=0x40 age=14h55m) si[0]=0x7f0aa402e600 (state=EST flags=0x204048 endp0=APPCTX:0x7f0aa402df90 exp= et=0x000 sub=0) si[1]=0x7f0aa402e658 (state=EST flags=0x2000d8 endp1=CS:0x7f0aa4039400 exp= et=0x000 sub=1) app0=0x7f0aa402df90 st0=7 st1=0 st2=0 applet= tmask=0x40 nice=0 calls=206100614 rate=3160 cpu=0 lat=0 co1=0x7f0aa4039250 ctrl=tcpv4 xprt=RAW mux=PASS data=STRM target=PROXY:0x564c0ff53760 flags=0x10003300 fd=430 fd.state=10121 updt=0 fd.tmask=0x40 cs=0x7f0aa4039400 csf=0x8200 ctx=(nil) req=0x7f0aa402e2d0 (f=0x848202 an=0x0 pipe=0 tofwd=-1 total=94919611
Re: Peers using heavily single cpu core
Hi Maciej, On Fri, Mar 04, 2022 at 01:10:37PM +0100, Maciej Zdeb wrote: > Hi, > > I'm experiencing high CPU usage on a single core, idle drops below 40% > while other cores are at 80% idle. Peers cluster is quite big (12 servers, > each server running 12 threads) and is used for synchronization of 3 > stick-tables of 1 million entries size. > > Is peers protocol single threaded and high usage on single core is expected > or am I experiencing some kind of bug? I'll keep digging to be sure nothing > else from my configuration is causing the issue. Almost. More precisely, each peers connection runs on a single thread at once (like any connection). Some such connections may experience heavy protocol parsing so it may be possible that sometimes you end up with an unbalance number of connections on threads. It's tricky though, because we could imagine a mechanism to try to evenly spread the outgoing peers connection on threads but the incoming ones will land where they land. That's something you can check with "show sess" and/or "show fd", looking for those related to your peers and checking their thread_mask. If you find two on the same thread (same thread_mask), you can shut one of them down using "shutdown session " and it will reconnect, possibly on another thread. That could confirm that it's the root cause of the problem you're experiencing. I'm wondering if we shouldn't introduce a notion of "heavy connection" like we already have heavy tasks for the scheduler. These ones would be balanced differently from others. Usually they're long-lived and can eat massive amounts of CPU so it would make sense. The only ones I'm thinking about are the peers ones but the concept would be more portable than focusing on peers. Usually such long connections are not refreshed often so we probably prefer to spend time carefully picking the best thread rather than saving 200ns of processing and having to pay it for the whole connection's life time. Willy
Peers using heavily single cpu core
Hi, I'm experiencing high CPU usage on a single core, idle drops below 40% while other cores are at 80% idle. Peers cluster is quite big (12 servers, each server running 12 threads) and is used for synchronization of 3 stick-tables of 1 million entries size. Is peers protocol single threaded and high usage on single core is expected or am I experiencing some kind of bug? I'll keep digging to be sure nothing else from my configuration is causing the issue. haproxy -v HAProxy version 2.4.14-eaa786f 2022/02/25 - https://haproxy.org/ Status: long-term supported branch - will stop receiving fixes around Q2 2026. Known bugs: http://www.haproxy.org/bugs/bugs-2.4.14.html Running on: Linux 4.15.0-154-generic #161-Ubuntu SMP Fri Jul 30 13:04:17 UTC 2021 x86_64 Kind regards, Maciej Zdeb
