Re: kernel: sonewconn: pcb 0xfffff8002b255a00 (local:/var/run/devd.seqpacket.pipe): Listen queue overflow: 1 already in queue awaiting acceptance (60 occurrences), ?

2023-07-18 Thread Graham Perrin 

On 21/06/2023 01:29, Graham Perrin wrote:


… Thanks, people.

A few hours ago I took a hint from one of the pages linked from 
, added a 
line to my sysctl.conf(5).


…
% grep ipc /etc/sysctl.conf
kern.ipc.soacceptqueue=256
%


I found no improvement from that. IIRC I also tried 512.



OpenPGP_signature
Description: OpenPGP digital signature

Re: kernel: sonewconn: pcb 0xfffff8002b255a00 (local:/var/run/devd.seqpacket.pipe): Listen queue overflow: 1 already in queue awaiting acceptance (60 occurrences), ?

2023-06-21 Thread Alexander Leidinger 

Quoting Gary Jennejohn  (from Tue, 20 Jun 2023 14:41:41 +):


On Tue, 20 Jun 2023 12:04:13 +0200
Alexander Leidinger  wrote:



"listen X backlog=y" and "sysctl kern.ipx.somaxconn=X" for FreeBSD



On my FreeBSD14 system these things are all under kern.ipc.


Typo on my side... it was supposed to read ipc, not ipx.

Bye,
Alexander.

--
http://www.Leidinger.net alexan...@leidinger.net: PGP 0x8F31830F9F2772BF
http://www.FreeBSD.orgnetch...@freebsd.org  : PGP 0x8F31830F9F2772BF


pgpTjpqaetzBB.pgp
Description: Digitale PGP-Signatur

Re: kernel: sonewconn: pcb 0xfffff8002b255a00 (local:/var/run/devd.seqpacket.pipe): Listen queue overflow: 1 already in queue awaiting acceptance (60 occurrences), ?

2023-06-20 Thread Graham Perrin 

On 20/06/2023 15:41, Gary Jennejohn wrote:

On Tue, 20 Jun 2023 12:04:13 +0200
Alexander Leidinger  wrote:


Quoting Gary Jennejohn  (from Tue, 20 Jun 2023 07:41:08 +):


…

In other words the software listening on it didn't process the request
fast enough and a backlog piled up (e.g apache ListenBacklog or nginx
"listen X backlog=y" and "sysctl kern.ipx.somaxconn=X" for FreeBSD
itself). You may need faster hardware, more processes/threads to
handle the traffic, or configure your software to do less to produce
the same result (e.g. no real-time DNS resolution in the logging of a
webserver or increasing the amount of allowed items in the backlog).
If you can change the software, there's also the possibility to switch
from blocking sockets to non-blocking sockets (to not have the
select/accept loop block / run into contention) or kqueue.


On my FreeBSD14 system these things are all under kern.ipc.

maxconn seems to be an alias for soacceptqueue, which is set to 128 on
my machine.

However, the software he's using may have set backlog to 1.  Hard to say
based on the trace he provided.


Thanks, people.

A few hours ago I took a hint from one of the pages linked from 
, added a 
line to my sysctl.conf(5).



% sysctl kern.ipc.soacceptqueue
kern.ipc.soacceptqueue: 256
% sysctl kern.ipc.maxconn
sysctl: unknown oid 'kern.ipc.maxconn'
% sysctl kern.ipc.somaxconn
kern.ipc.somaxconn: 256
% grep ipc /etc/sysctl.conf
kern.ipc.soacceptqueue=256
%


I began wondering about the sonewconn lines in relation to 
 for 
net-mgmt/netdata, although I don't know whether there's a direct 
relationship.


% netstat -Lan | grep netdata
unix  0/0/128  /tmp/netdata-ipc
%

My use of Netdata is local-only.



OpenPGP_signature
Description: OpenPGP digital signature

Re: kernel: sonewconn: pcb 0xfffff8002b255a00 (local:/var/run/devd.seqpacket.pipe): Listen queue overflow: 1 already in queue awaiting acceptance (60 occurrences), ?

2023-06-20 Thread Gary Jennejohn 
On Tue, 20 Jun 2023 12:04:13 +0200
Alexander Leidinger  wrote:

> Quoting Gary Jennejohn  (from Tue, 20 Jun 2023 07:41:08 +):
>
> > On Tue, 20 Jun 2023 06:25:05 +0100
> > Graham Perrin  wrote:
> >
> >> Please, what's the meaning of the sonewconn lines?
> >>
> >
> > sonewconn is described in socket(9).  Below a copy/paste of the description
> > from socket(9):
> >
> >  Protocol implementations can use sonewconn() to create a socket and
> >  attach protocol state to that socket.  This can be used to create new
> >  sockets available for soaccept() on a listen socket.  The
> > returned socket
> >  has a reference count of zero.
> >
> > Apparently there was already a listen socket in the queue which had not been
> > consumed by soaccept() when a new sonewconn() call was made.
> >
> > Anyway, that's my understanding.  Might be wrong.
>
> In other words the software listening on it didn't process the request
> fast enough and a backlog piled up (e.g apache ListenBacklog or nginx
> "listen X backlog=y" and "sysctl kern.ipx.somaxconn=X" for FreeBSD
> itself). You may need faster hardware, more processes/threads to
> handle the traffic, or configure your software to do less to produce
> the same result (e.g. no real-time DNS resolution in the logging of a
> webserver or increasing the amount of allowed items in the backlog).
> If you can change the software, there's also the possibility to switch
> from blocking sockets to non-blocking sockets (to not have the
> select/accept loop block / run into contention) or kqueue.
>

On my FreeBSD14 system these things are all under kern.ipc.

maxconn seems to be an alias for soacceptqueue, which is set to 128 on
my machine.

However, the software he's using may have set backlog to 1.  Hard to say
based on the trace he provided.

--
Gary Jennejohn

Re: kernel: sonewconn: pcb 0xfffff8002b255a00 (local:/var/run/devd.seqpacket.pipe): Listen queue overflow: 1 already in queue awaiting acceptance (60 occurrences), ?

2023-06-20 Thread Alexander Leidinger 


Quoting Gary Jennejohn  (from Tue, 20 Jun 2023 07:41:08 +):


On Tue, 20 Jun 2023 06:25:05 +0100
Graham Perrin  wrote:


Please, what's the meaning of the sonewconn lines?



sonewconn is described in socket(9).  Below a copy/paste of the description
from socket(9):

 Protocol implementations can use sonewconn() to create a socket and
 attach protocol state to that socket.  This can be used to create new
 sockets available for soaccept() on a listen socket.  The  
returned socket

 has a reference count of zero.

Apparently there was already a listen socket in the queue which had not been
consumed by soaccept() when a new sonewconn() call was made.

Anyway, that's my understanding.  Might be wrong.


In other words the software listening on it didn't process the request  
fast enough and a backlog piled up (e.g apache ListenBacklog or nginx  
"listen X backlog=y" and "sysctl kern.ipx.somaxconn=X" for FreeBSD  
itself). You may need faster hardware, more processes/threads to  
handle the traffic, or configure your software to do less to produce  
the same result (e.g. no real-time DNS resolution in the logging of a  
webserver or increasing the amount of allowed items in the backlog).  
If you can change the software, there's also the possibility to switch  
from blocking sockets to non-blocking sockets (to not have the  
select/accept loop block / run into contention) or kqueue.


Bye,
Alexander.

--
http://www.Leidinger.net alexan...@leidinger.net: PGP 0x8F31830F9F2772BF
http://www.FreeBSD.orgnetch...@freebsd.org  : PGP 0x8F31830F9F2772BF


pgpAjQQlBmAmQ.pgp
Description: Digitale PGP-Signatur

Re: kernel: sonewconn: pcb 0xfffff8002b255a00 (local:/var/run/devd.seqpacket.pipe): Listen queue overflow: 1 already in queue awaiting acceptance (60 occurrences), ?

2023-06-20 Thread Gary Jennejohn 
On Tue, 20 Jun 2023 06:25:05 +0100
Graham Perrin  wrote:

> Please, what's the meaning of the sonewconn lines?
>

sonewconn is described in socket(9).  Below a copy/paste of the description
from socket(9):

 Protocol implementations can use sonewconn() to create a socket and
 attach protocol state to that socket.  This can be used to create new
 sockets available for soaccept() on a listen socket.  The returned socket
 has a reference count of zero.

Apparently there was already a listen socket in the queue which had not been
consumed by soaccept() when a new sonewconn() call was made.

Anyway, that's my understanding.  Might be wrong.

> % tail -f -n 0 /var/log/messages
> Jun 20 04:08:41 mowa219-gjp4-8570p-freebsd su[4159]: grahamperrin to
> root on /dev/pts/6
> Jun 20 04:12:20 mowa219-gjp4-8570p-freebsd apps.plugin[3178]: Cannot
> fetch process 4501 command line (command 'sysctl')
> Jun 20 04:30:25 mowa219-gjp4-8570p-freebsd kernel: lock order reversal:
> Jun 20 04:30:25 mowa219-gjp4-8570p-freebsd kernel:  1st
> 0xfe0103db30e0 filedesc structure (filedesc structure, sx) @
> /usr/src/sys/kern/kern_descrip.c:1397
> Jun 20 04:30:25 mowa219-gjp4-8570p-freebsd kernel:  2nd
> 0xf80001df8930 devfs (devfs, lockmgr) @
> /usr/src/sys/kern/vfs_subr.c:6244
> Jun 20 04:30:25 mowa219-gjp4-8570p-freebsd kernel: lock order filedesc
> structure -> devfs attempted at:
> Jun 20 04:30:25 mowa219-gjp4-8570p-freebsd kernel: #0 0x80bc4573
> at witness_checkorder+0xbb3
> Jun 20 04:30:25 mowa219-gjp4-8570p-freebsd kernel: #1 0x80b21265
> at lockmgr_xlock+0x55
> Jun 20 04:30:25 mowa219-gjp4-8570p-freebsd kernel: #2 0x80c5e4b4
> at _vn_lock+0x54
> Jun 20 04:30:25 mowa219-gjp4-8570p-freebsd kernel: #3 0x80afd13b
> at knlist_remove_kq+0x8b
> Jun 20 04:30:25 mowa219-gjp4-8570p-freebsd kernel: #4 0x80c517d4
> at filt_vfsdetach+0x24
> Jun 20 04:30:25 mowa219-gjp4-8570p-freebsd kernel: #5 0x80afdad0
> at knote_fdclose+0x180
> Jun 20 04:30:25 mowa219-gjp4-8570p-freebsd kernel: #6 0x80af4f2c
> at closefp_impl+0x9c
> Jun 20 04:30:25 mowa219-gjp4-8570p-freebsd kernel: #7 0x8104d8e0
> at amd64_syscall+0x140
> Jun 20 04:30:25 mowa219-gjp4-8570p-freebsd kernel: #8 0x81020b3b
> at fast_syscall_common+0xf8
> Jun 20 04:34:26 mowa219-gjp4-8570p-freebsd apps.plugin[3178]: heartbeat
> clock: woke up 1346787 microseconds later than expected (can be due to
> system load or the CLOCK_REALTIME set to the future).
> Jun 20 04:35:38 mowa219-gjp4-8570p-freebsd apps.plugin[3178]: Cannot
> fetch process 6471 command line (command 'sysctl')
> Jun 20 04:55:30 mowa219-gjp4-8570p-freebsd apps.plugin[3178]: Cannot
> fetch process 7697 command line (command 'sysctl')
> Jun 20 05:07:16 mowa219-gjp4-8570p-freebsd apps.plugin[3178]: Cannot
> fetch process 8416 command line (command 'sysctl')
> Jun 20 05:17:42 mowa219-gjp4-8570p-freebsd kernel: sonewconn: pcb
> 0xf8002b255a00 (local:/var/run/devd.seqpacket.pipe): Listen queue
> overflow: 1 already in queue awaiting acceptance (1 occurrences), euid
> 0, rgid
> 0, jail 0
> Jun 20 05:18:42 mowa219-gjp4-8570p-freebsd kernel: sonewconn: pcb
> 0xf8002b255a00 (local:/var/run/devd.seqpacket.pipe): Listen queue
> overflow: 1 already in queue awaiting acceptance (60 occurrences), euid
> 0, rgi
> d 0, jail 0
> Jun 20 05:18:56 mowa219-gjp4-8570p-freebsd apps.plugin[3178]: Cannot
> fetch process 9359 command line (command 'git')
> Jun 20 05:19:42 mowa219-gjp4-8570p-freebsd kernel: sonewconn: pcb
> 0xf8002b255a00 (local:/var/run/devd.seqpacket.pipe): Listen queue
> overflow: 1 already in queue awaiting acceptance (60 occurrences), euid
> 0, rgi
> d 0, jail 0
> Jun 20 05:20:03 mowa219-gjp4-8570p-freebsd apps.plugin[3178]: Cannot
> fetch process 9602 command line (command 'cat')
> Jun 20 05:20:43 mowa219-gjp4-8570p-freebsd kernel: sonewconn: pcb
> 0xf8002b255a00 (local:/var/run/devd.seqpacket.pipe): Listen queue
> overflow: 1 already in queue awaiting acceptance (60 occurrences), euid
> 0, rgi
> d 0, jail 0
> ^C
> % sudo sysctl kern.sched.pick_short=1
> grahamperrin's password:
> kern.sched.pick_short: 0 -> 1
> % sudo sysctl kern.sched.preempt_thresh=224
> kern.sched.preempt_thresh: 48 -> 224
> % uptime
> 6:23a.m.  up  2:24, 7 users, load averages: 0.78, 1.24, 1.34
> % uname -aKU
> FreeBSD mowa219-gjp4-8570p-freebsd 14.0-CURRENT FreeBSD 14.0-CURRENT #7
> main-n263630-ab3e6234ab6e-dirty: Sun Jun 18 14:56:48 BST 2023
> grahamperrin@mowa219-gjp4-8570p-freebsd:/usr/obj/usr/src/amd64.amd64/sys/GEN
> ERIC amd64 1400090 1400090
> %
>
>


--
Gary Jennejohn

kernel: sonewconn: pcb 0xfffff8002b255a00 (local:/var/run/devd.seqpacket.pipe): Listen queue overflow: 1 already in queue awaiting acceptance (60 occurrences), …

2023-06-19 Thread Graham Perrin 

Please, what's the meaning of the sonewconn lines?


% tail -f -n 0 /var/log/messages
Jun 20 04:08:41 mowa219-gjp4-8570p-freebsd su[4159]: grahamperrin to 
root on /dev/pts/6
Jun 20 04:12:20 mowa219-gjp4-8570p-freebsd apps.plugin[3178]: Cannot 
fetch process 4501 command line (command 'sysctl')

Jun 20 04:30:25 mowa219-gjp4-8570p-freebsd kernel: lock order reversal:
Jun 20 04:30:25 mowa219-gjp4-8570p-freebsd kernel:  1st 
0xfe0103db30e0 filedesc structure (filedesc structure, sx) @ 
/usr/src/sys/kern/kern_descrip.c:1397
Jun 20 04:30:25 mowa219-gjp4-8570p-freebsd kernel:  2nd 
0xf80001df8930 devfs (devfs, lockmgr) @ 
/usr/src/sys/kern/vfs_subr.c:6244
Jun 20 04:30:25 mowa219-gjp4-8570p-freebsd kernel: lock order filedesc 
structure -> devfs attempted at:
Jun 20 04:30:25 mowa219-gjp4-8570p-freebsd kernel: #0 0x80bc4573 
at witness_checkorder+0xbb3
Jun 20 04:30:25 mowa219-gjp4-8570p-freebsd kernel: #1 0x80b21265 
at lockmgr_xlock+0x55
Jun 20 04:30:25 mowa219-gjp4-8570p-freebsd kernel: #2 0x80c5e4b4 
at _vn_lock+0x54
Jun 20 04:30:25 mowa219-gjp4-8570p-freebsd kernel: #3 0x80afd13b 
at knlist_remove_kq+0x8b
Jun 20 04:30:25 mowa219-gjp4-8570p-freebsd kernel: #4 0x80c517d4 
at filt_vfsdetach+0x24
Jun 20 04:30:25 mowa219-gjp4-8570p-freebsd kernel: #5 0x80afdad0 
at knote_fdclose+0x180
Jun 20 04:30:25 mowa219-gjp4-8570p-freebsd kernel: #6 0x80af4f2c 
at closefp_impl+0x9c
Jun 20 04:30:25 mowa219-gjp4-8570p-freebsd kernel: #7 0x8104d8e0 
at amd64_syscall+0x140
Jun 20 04:30:25 mowa219-gjp4-8570p-freebsd kernel: #8 0x81020b3b 
at fast_syscall_common+0xf8
Jun 20 04:34:26 mowa219-gjp4-8570p-freebsd apps.plugin[3178]: heartbeat 
clock: woke up 1346787 microseconds later than expected (can be due to 
system load or the CLOCK_REALTIME set to the future).
Jun 20 04:35:38 mowa219-gjp4-8570p-freebsd apps.plugin[3178]: Cannot 
fetch process 6471 command line (command 'sysctl')
Jun 20 04:55:30 mowa219-gjp4-8570p-freebsd apps.plugin[3178]: Cannot 
fetch process 7697 command line (command 'sysctl')
Jun 20 05:07:16 mowa219-gjp4-8570p-freebsd apps.plugin[3178]: Cannot 
fetch process 8416 command line (command 'sysctl')
Jun 20 05:17:42 mowa219-gjp4-8570p-freebsd kernel: sonewconn: pcb 
0xf8002b255a00 (local:/var/run/devd.seqpacket.pipe): Listen queue 
overflow: 1 already in queue awaiting acceptance (1 occurrences), euid 
0, rgid

0, jail 0
Jun 20 05:18:42 mowa219-gjp4-8570p-freebsd kernel: sonewconn: pcb 
0xf8002b255a00 (local:/var/run/devd.seqpacket.pipe): Listen queue 
overflow: 1 already in queue awaiting acceptance (60 occurrences), euid 
0, rgi

d 0, jail 0
Jun 20 05:18:56 mowa219-gjp4-8570p-freebsd apps.plugin[3178]: Cannot 
fetch process 9359 command line (command 'git')
Jun 20 05:19:42 mowa219-gjp4-8570p-freebsd kernel: sonewconn: pcb 
0xf8002b255a00 (local:/var/run/devd.seqpacket.pipe): Listen queue 
overflow: 1 already in queue awaiting acceptance (60 occurrences), euid 
0, rgi

d 0, jail 0
Jun 20 05:20:03 mowa219-gjp4-8570p-freebsd apps.plugin[3178]: Cannot 
fetch process 9602 command line (command 'cat')
Jun 20 05:20:43 mowa219-gjp4-8570p-freebsd kernel: sonewconn: pcb 
0xf8002b255a00 (local:/var/run/devd.seqpacket.pipe): Listen queue 
overflow: 1 already in queue awaiting acceptance (60 occurrences), euid 
0, rgi

d 0, jail 0
^C
% sudo sysctl kern.sched.pick_short=1
grahamperrin's password:
kern.sched.pick_short: 0 -> 1
% sudo sysctl kern.sched.preempt_thresh=224
kern.sched.preempt_thresh: 48 -> 224
% uptime
6:23a.m.  up  2:24, 7 users, load averages: 0.78, 1.24, 1.34
% uname -aKU
FreeBSD mowa219-gjp4-8570p-freebsd 14.0-CURRENT FreeBSD 14.0-CURRENT #7 
main-n263630-ab3e6234ab6e-dirty: Sun Jun 18 14:56:48 BST 2023 
grahamperrin@mowa219-gjp4-8570p-freebsd:/usr/obj/usr/src/amd64.amd64/sys/GEN

ERIC amd64 1400090 1400090
%




OpenPGP_signature
Description: OpenPGP digital signature