Hi Willy,
did you find anything?
I've made some "session shutdown" tests. With simple configuration
(attached) it is very easy to reproduce a situation when "shutdown session"
works only for the first time.
1. Kill haproxy to have a clean start:
root@hap-rnd-1a:/etc/haproxy# killall haproxy
2. Show current sessions:
root@hap-rnd-1a:/etc/haproxy# echo "show sess" | socat
unix-connect:/var/run/haproxy/haproxy1.sock stdio
0x562291719560: proto=tcpv4 src=10.0.0.3:5276 fe=hap-rnd-1a be=hap-rnd-1a
srv= ts=00 epoch=0 age=4s calls=1 rate=0 cpu=0 lat=0
rq[f=c48202h,i=0,an=00h,rx=,wx=,ax=] rp[f=80048202h,i=0,an=00h,rx=,wx=,ax=]
s0=[8,200048h,fd=29,ex=] s1=[8,204058h,fd=-1,ex=] exp=
0x562291719fa0: proto=unix_stream src=unix:1 fe=GLOBAL be= srv=
ts=00 epoch=0 age=0s calls=2 rate=2 cpu=0 lat=0
rq[f=c0c020h,i=0,an=00h,rx=,wx=,ax=] rp[f=80008002h,i=0,an=00h,rx=,wx=,ax=]
s0=[8,280008h,fd=30,ex=] s1=[8,204018h,fd=-1,ex=] exp=30s
3. Kill peer session 0x562291719560
root@hap-rnd-1a:/etc/haproxy# echo "shutdown session 0x562291719560" |
socat unix-connect:/var/run/haproxy/haproxy1.sock stdio
4. Show current sessions - confirm peer connection shutdown
root@hap-rnd-1a:/etc/haproxy# echo "show sess" | socat
unix-connect:/var/run/haproxy/haproxy1.sock stdio
0x7f2628026220: proto=unix_stream src=unix:1 fe=GLOBAL be= srv=
ts=00 epoch=0x1 age=0s calls=1 rate=1 cpu=0 lat=0
rq[f=c08000h,i=0,an=00h,rx=,wx=,ax=] rp[f=80008000h,i=0,an=00h,rx=,wx=,ax=]
s0=[8,240008h,fd=29,ex=] s1=[8,204018h,fd=-1,ex=] exp=
5. Show current sessions - confirm new peer connection (note that new
connection has same id 0x7f2628026220 as unix_stream in previous output,
but it is probably by chance)
root@hap-rnd-1a:/etc/haproxy# echo "show sess" | socat
unix-connect:/var/run/haproxy/haproxy1.sock stdio
0x7f2628026220: proto=tcpv4 src=10.0.0.3:5288 fe=hap-rnd-1a be=hap-rnd-1a
srv= ts=00 epoch=0x2 age=3s calls=1 rate=0 cpu=0 lat=0
rq[f=c48200h,i=0,an=00h,rx=,wx=,ax=] rp[f=80048202h,i=0,an=00h,rx=,wx=,ax=]
s0=[8,200048h,fd=29,ex=] s1=[8,204058h,fd=-1,ex=] exp=
0x7f263c026220: proto=unix_stream src=unix:1 fe=GLOBAL be= srv=
ts=00 epoch=0x2 age=0s calls=1 rate=1 cpu=0 lat=0
rq[f=c08000h,i=0,an=00h,rx=,wx=,ax=] rp[f=80008002h,i=0,an=00h,rx=,wx=,ax=]
s0=[8,240008h,fd=30,ex=] s1=[8,204018h,fd=-1,ex=] exp=
6. Again kill peer session 0x7f2628026220
root@hap-rnd-1a:/etc/haproxy# echo "shutdown session 0x7f2628026220" |
socat unix-connect:/var/run/haproxy/haproxy1.sock stdio
7. Show current sessions - note that 0x7f2628026220 was not killed in the
previous step
root@hap-rnd-1a:/etc/haproxy# echo "show sess" | socat
unix-connect:/var/run/haproxy/haproxy1.sock stdio
0x7f267c026a60: proto=unix_stream src=unix:1 fe=GLOBAL be= srv=
ts=00 epoch=0x3 age=0s calls=1 rate=1 cpu=0 lat=0
rq[f=c08000h,i=0,an=00h,rx=,wx=,ax=] rp[f=80008000h,i=0,an=00h,rx=,wx=,ax=]
s0=[8,240008h,fd=30,ex=] s1=[8,204018h,fd=-1,ex=] exp=
0x7f2628026220: proto=tcpv4 src=10.0.0.3:5288 fe=hap-rnd-1a be=hap-rnd-1a
srv= ts=00 epoch=0x2 age=17s calls=1 rate=0 cpu=0 lat=0
rq[f=c48202h,i=0,an=00h,rx=,wx=,ax=] rp[f=80048202h,i=0,an=00h,rx=,wx=,ax=]
s0=[8,200048h,fd=29,ex=] s1=[8,204058h,fd=-1,ex=] exp=
8. Kill peer session 0x7f2628026220
root@hap-rnd-1a:/etc/haproxy# echo "shutdown session 0x7f2628026220" |
socat unix-connect:/var/run/haproxy/haproxy1.sock stdio
9. Show session - again no effect
root@hap-rnd-1a:/etc/haproxy# echo "show sess" | socat
unix-connect:/var/run/haproxy/haproxy1.sock stdio
0x7f2628026220: proto=tcpv4 src=10.0.0.3:5288 fe=hap-rnd-1a be=hap-rnd-1a
srv= ts=00 epoch=0x2 age=22s calls=1 rate=0 cpu=0 lat=0
rq[f=c48202h,i=0,an=00h,rx=,wx=,ax=] rp[f=80048202h,i=0,an=00h,rx=,wx=,ax=]
s0=[8,200048h,fd=29,ex=] s1=[8,204058h,fd=-1,ex=] exp=
0x7f261c026220: proto=unix_stream src=unix:1 fe=GLOBAL be= srv=
ts=00 epoch=0x4 age=0s calls=1 rate=1 cpu=0 lat=0
rq[f=c08000h,i=0,an=00h,rx=,wx=,ax=] rp[f=80008002h,i=0,an=00h,rx=,wx=,ax=]
s0=[8,240008h,fd=30,ex=] s1=[8,204018h,fd=-1,ex=] exp=
Kind regards,
sob., 12 mar 2022 o 00:20 Willy Tarreau napisał(a):
> On Fri, Mar 11, 2022 at 10:19:09PM +0100, Maciej Zdeb wrote:
> > Hi Willy,
> >
> > Thank you for such useful info! I've checked the worst HAProxy nodes and
> on
> > every such node all outgoing peers connections are run on the same
> thread:
> (...)
> Indeed. I was pretty sure we were setting them to any thread on creation
> but maybe I'm wrong, I'll have to recheck.
>
> > On one node I was able to rebalance it, but on the node above (and other
> > nodes) I'm not able to shutdown the sessions:
> (...)
> > echo "shutdown session 0x7f0aa402e2c0" | socat
> > unix-connect:/var/run/haproxy.sock stdio
> >
> > echo "show sess 0x7f0aa402e2c0" | socat
> unix-connect:/var/run/haproxy.sock
> > stdio
> > 0x7f0aa402e2c0: [11/Mar/2022:07:17:02.313221] id=0 proto=?
> (...)
>
> That's not expected, another thing I'll have to check.
>
> Thanks for testing. I'll put that in pause for the week-end, though :-)
>
> cheers,
> Willy
>
global
chroot
