On Tue, Mar 07, 2023 at 08:36:24AM +0000, Stuart Henderson wrote: > On 2023/03/07 07:10, Tom Smyth wrote: > > I m running smokeping fcgi and rrdcached ontop of OpenbSD, to smokeping > > about 150 devces > > the page load times can take 30 seconds to 1 minute, > > is there any way to speed this up. > > > > im running 7.2 OpenBSD on amd64 vm on top of an SSD array > > > > any tips tricks welccome ... > > One quick thing to try is updating to -current, I made some changes to > the rrdtool port which may possibly help a little. > > Check that smokeping is actually using rrdcached (watch top while > opening a page) - the pkg-readme only gives instructions for passing the > required fastcgi variable through for nginx, I don't know how to do that > for httpd (or whether it's actually possible). > > Other than that, rrdtool/rrdcached is just slow on OpenBSD. If it's > anything like mine you'll see high cpu spin % in top while it's busy. > You can try changing the number of cores in the VM - if you've given it > lots of cores try *reducing* it a bit. To pick a number out of the air > I'd suggest probably 4-6. (mine is bare metal and I can't drop the > number short of kernel hacks to set more cores offline). > > You can hide the slowness at the loss of dynamic functionality in the > web interface by pre rendering the html/graphs from a cron job rather > than using the fastcgi (see the pkg-readme). But other than the above > I'm out of ideas to actually make it run faster. > > (If anyone interested in poking at kernel locks would like flamegraphs > from my monitoring box - librenms/smokeping/icinga2/mariadb with lots > of rrdtool/rrdcached - let me know. Spin is pretty brutal.)
No need to collect flamegraphs, the issue is massive contention on the kernel lock because of high IO load. I see similar behaviour with iogen. Currently competing read and write calls clash with the async buffer handling which also requires the kernel lock to finish their work. So more concurrency makes it worse. Fixing this is a major task. -- :wq Claudio
