Hi Willy, did you find anything? I've made some "session shutdown" tests. With simple configuration (attached) it is very easy to reproduce a situation when "shutdown session" works only for the first time.
1. Kill haproxy to have a clean start: root@hap-rnd-1a:/etc/haproxy# killall haproxy 2. Show current sessions: root@hap-rnd-1a:/etc/haproxy# echo "show sess" | socat unix-connect:/var/run/haproxy/haproxy1.sock stdio 0x562291719560: proto=tcpv4 src=10.0.0.3:5276 fe=hap-rnd-1a be=hap-rnd-1a srv=<none> ts=00 epoch=0 age=4s calls=1 rate=0 cpu=0 lat=0 rq[f=c48202h,i=0,an=00h,rx=,wx=,ax=] rp[f=80048202h,i=0,an=00h,rx=,wx=,ax=] s0=[8,200048h,fd=29,ex=] s1=[8,204058h,fd=-1,ex=] exp= 0x562291719fa0: proto=unix_stream src=unix:1 fe=GLOBAL be=<NONE> srv=<none> ts=00 epoch=0 age=0s calls=2 rate=2 cpu=0 lat=0 rq[f=c0c020h,i=0,an=00h,rx=,wx=,ax=] rp[f=80008002h,i=0,an=00h,rx=,wx=,ax=] s0=[8,280008h,fd=30,ex=] s1=[8,204018h,fd=-1,ex=] exp=30s 3. Kill peer session 0x562291719560 root@hap-rnd-1a:/etc/haproxy# echo "shutdown session 0x562291719560" | socat unix-connect:/var/run/haproxy/haproxy1.sock stdio 4. Show current sessions - confirm peer connection shutdown root@hap-rnd-1a:/etc/haproxy# echo "show sess" | socat unix-connect:/var/run/haproxy/haproxy1.sock stdio 0x7f2628026220: proto=unix_stream src=unix:1 fe=GLOBAL be=<NONE> srv=<none> ts=00 epoch=0x1 age=0s calls=1 rate=1 cpu=0 lat=0 rq[f=c08000h,i=0,an=00h,rx=,wx=,ax=] rp[f=80008000h,i=0,an=00h,rx=,wx=,ax=] s0=[8,240008h,fd=29,ex=] s1=[8,204018h,fd=-1,ex=] exp= 5. Show current sessions - confirm new peer connection (note that new connection has same id 0x7f2628026220 as unix_stream in previous output, but it is probably by chance) root@hap-rnd-1a:/etc/haproxy# echo "show sess" | socat unix-connect:/var/run/haproxy/haproxy1.sock stdio 0x7f2628026220: proto=tcpv4 src=10.0.0.3:5288 fe=hap-rnd-1a be=hap-rnd-1a srv=<none> ts=00 epoch=0x2 age=3s calls=1 rate=0 cpu=0 lat=0 rq[f=c48200h,i=0,an=00h,rx=,wx=,ax=] rp[f=80048202h,i=0,an=00h,rx=,wx=,ax=] s0=[8,200048h,fd=29,ex=] s1=[8,204058h,fd=-1,ex=] exp= 0x7f263c026220: proto=unix_stream src=unix:1 fe=GLOBAL be=<NONE> srv=<none> ts=00 epoch=0x2 age=0s calls=1 rate=1 cpu=0 lat=0 rq[f=c08000h,i=0,an=00h,rx=,wx=,ax=] rp[f=80008002h,i=0,an=00h,rx=,wx=,ax=] s0=[8,240008h,fd=30,ex=] s1=[8,204018h,fd=-1,ex=] exp= 6. Again kill peer session 0x7f2628026220 root@hap-rnd-1a:/etc/haproxy# echo "shutdown session 0x7f2628026220" | socat unix-connect:/var/run/haproxy/haproxy1.sock stdio 7. Show current sessions - note that 0x7f2628026220 was not killed in the previous step root@hap-rnd-1a:/etc/haproxy# echo "show sess" | socat unix-connect:/var/run/haproxy/haproxy1.sock stdio 0x7f267c026a60: proto=unix_stream src=unix:1 fe=GLOBAL be=<NONE> srv=<none> ts=00 epoch=0x3 age=0s calls=1 rate=1 cpu=0 lat=0 rq[f=c08000h,i=0,an=00h,rx=,wx=,ax=] rp[f=80008000h,i=0,an=00h,rx=,wx=,ax=] s0=[8,240008h,fd=30,ex=] s1=[8,204018h,fd=-1,ex=] exp= 0x7f2628026220: proto=tcpv4 src=10.0.0.3:5288 fe=hap-rnd-1a be=hap-rnd-1a srv=<none> ts=00 epoch=0x2 age=17s calls=1 rate=0 cpu=0 lat=0 rq[f=c48202h,i=0,an=00h,rx=,wx=,ax=] rp[f=80048202h,i=0,an=00h,rx=,wx=,ax=] s0=[8,200048h,fd=29,ex=] s1=[8,204058h,fd=-1,ex=] exp= 8. Kill peer session 0x7f2628026220 root@hap-rnd-1a:/etc/haproxy# echo "shutdown session 0x7f2628026220" | socat unix-connect:/var/run/haproxy/haproxy1.sock stdio 9. Show session - again no effect root@hap-rnd-1a:/etc/haproxy# echo "show sess" | socat unix-connect:/var/run/haproxy/haproxy1.sock stdio 0x7f2628026220: proto=tcpv4 src=10.0.0.3:5288 fe=hap-rnd-1a be=hap-rnd-1a srv=<none> ts=00 epoch=0x2 age=22s calls=1 rate=0 cpu=0 lat=0 rq[f=c48202h,i=0,an=00h,rx=,wx=,ax=] rp[f=80048202h,i=0,an=00h,rx=,wx=,ax=] s0=[8,200048h,fd=29,ex=] s1=[8,204058h,fd=-1,ex=] exp= 0x7f261c026220: proto=unix_stream src=unix:1 fe=GLOBAL be=<NONE> srv=<none> ts=00 epoch=0x4 age=0s calls=1 rate=1 cpu=0 lat=0 rq[f=c08000h,i=0,an=00h,rx=,wx=,ax=] rp[f=80008002h,i=0,an=00h,rx=,wx=,ax=] s0=[8,240008h,fd=30,ex=] s1=[8,204018h,fd=-1,ex=] exp= Kind regards, sob., 12 mar 2022 o 00:20 Willy Tarreau <w...@1wt.eu> napisaĆ(a): > On Fri, Mar 11, 2022 at 10:19:09PM +0100, Maciej Zdeb wrote: > > Hi Willy, > > > > Thank you for such useful info! I've checked the worst HAProxy nodes and > on > > every such node all outgoing peers connections are run on the same > thread: > (...) > Indeed. I was pretty sure we were setting them to any thread on creation > but maybe I'm wrong, I'll have to recheck. > > > On one node I was able to rebalance it, but on the node above (and other > > nodes) I'm not able to shutdown the sessions: > (...) > > echo "shutdown session 0x7f0aa402e2c0" | socat > > unix-connect:/var/run/haproxy.sock stdio > > > > echo "show sess 0x7f0aa402e2c0" | socat > unix-connect:/var/run/haproxy.sock > > stdio > > 0x7f0aa402e2c0: [11/Mar/2022:07:17:02.313221] id=0 proto=? > (...) > > That's not expected, another thing I'll have to check. > > Thanks for testing. I'll put that in pause for the week-end, though :-) > > cheers, > Willy >
global chroot /var/lib/haproxy stats socket /run/haproxy/haproxy1.sock mode 660 level admin process 1 stats timeout 30s user haproxy group haproxy daemon nbproc 1 nbthread 8 cpu-map auto:1/1-8 2-9 defaults mode http option allbackups peers dc5-rnd peer hap-rnd-1a.local 10.0.0.2:1024 peer hap-rnd-1b.local 10.0.0.3:1024 backend per_ip_rates stick-table type ip size 1m expire 10s store http_req_rate(10s) peers dc5-rnd