Disk accesses freeze for a lot of seconds
We have an OpenBSD 5.2 amd64 where every 5 minutes a few thousand of .rrd files from MRTG are written (actually, updated) to disk. The problem is that for a few seconds (15-20) every other access to the disk is totally blocked. So during those 15-20 seconds the access to the graphs is freezed! And this is really annoying for a graphs server... It's not a problem of CPU load (it's a quadruple core AMD Athlon II X4 630 Processor). Processes run smoothly, they freeze only when they try to access the disk. Disk is a normal SATA, and the partition is FFS with softdep. Is there anything (some system tuning?) I can do to get rid of the freezes, or at least to mitigate them? Thanks.
Re: Disk accesses freeze for a lot of seconds
On Sun, Jan 06, 2013 at 12:22:44PM +0100, Federico Giannici wrote: We have an OpenBSD 5.2 amd64 where every 5 minutes a few thousand of .rrd files from MRTG are written (actually, updated) to disk. The problem is that for a few seconds (15-20) every other access to the disk is totally blocked. So during those 15-20 seconds the access to the graphs is freezed! And this is really annoying for a graphs server... It's not a problem of CPU load (it's a quadruple core AMD Athlon II X4 630 Processor). Processes run smoothly, they freeze only when they try to access the disk. Disk is a normal SATA, and the partition is FFS with softdep. It's probably the bug in the buffer cache where the kernel would allow userland to queue up so many writes that eventually the kernel is starved out of buffers. Everything else (for example, read operations on your graph files) then sleeps until enough writes have been spilled out to disk. Is there anything (some system tuning?) I can do to get rid of the freezes, or at least to mitigate them? The best solution is an upgrade to -current where this has been fixed. See http://marc.info/?l=openbsd-cvsm=135231065926430w=2 and other related commits by Bob Beck. If you'd rather stick to 5.2 you can try turning off softdep. softdep delays some write operations so turning it off might help somewhat by allowing more read operations to interleave with write operations. While the bug was affecting -current I found that my systems where much more responsive with softdep turned off.
Re: Disk accesses freeze for a lot of seconds
You was right: turning off softdep made the freezes much shorter. Thanks. On 01/06/13 13:49, Stefan Sperling wrote: On Sun, Jan 06, 2013 at 12:22:44PM +0100, Federico Giannici wrote: We have an OpenBSD 5.2 amd64 where every 5 minutes a few thousand of .rrd files from MRTG are written (actually, updated) to disk. The problem is that for a few seconds (15-20) every other access to the disk is totally blocked. So during those 15-20 seconds the access to the graphs is freezed! And this is really annoying for a graphs server... It's not a problem of CPU load (it's a quadruple core AMD Athlon II X4 630 Processor). Processes run smoothly, they freeze only when they try to access the disk. Disk is a normal SATA, and the partition is FFS with softdep. It's probably the bug in the buffer cache where the kernel would allow userland to queue up so many writes that eventually the kernel is starved out of buffers. Everything else (for example, read operations on your graph files) then sleeps until enough writes have been spilled out to disk. Is there anything (some system tuning?) I can do to get rid of the freezes, or at least to mitigate them? The best solution is an upgrade to -current where this has been fixed. See http://marc.info/?l=openbsd-cvsm=135231065926430w=2 and other related commits by Bob Beck. If you'd rather stick to 5.2 you can try turning off softdep. softdep delays some write operations so turning it off might help somewhat by allowing more read operations to interleave with write operations. While the bug was affecting -current I found that my systems where much more responsive with softdep turned off.
Re: Disk accesses freeze for a lot of seconds
I got same problem with squid when squid exit normally (/etc/rc.d/squid stop), when mass squid disk cache is written, there is a one min freeze on the server. (OpenBSD 5.2). The problem was also here under OpenBSD 5.1. CPU is also OK (10% of a big xeon quad). But for me softdeps aren't activated. The temporary solution i used, kill -9 squid process when stop/restart is done. -- Cordialement, Loïc BLOT, UNIX systems, security and network expert http://www.unix-experience.fr Le dimanche 06 janvier 2013 à 16:08 +0100, Federico Giannici a écrit : You was right: turning off softdep made the freezes much shorter. Thanks. On 01/06/13 13:49, Stefan Sperling wrote: On Sun, Jan 06, 2013 at 12:22:44PM +0100, Federico Giannici wrote: We have an OpenBSD 5.2 amd64 where every 5 minutes a few thousand of .rrd files from MRTG are written (actually, updated) to disk. The problem is that for a few seconds (15-20) every other access to the disk is totally blocked. So during those 15-20 seconds the access to the graphs is freezed! And this is really annoying for a graphs server... It's not a problem of CPU load (it's a quadruple core AMD Athlon II X4 630 Processor). Processes run smoothly, they freeze only when they try to access the disk. Disk is a normal SATA, and the partition is FFS with softdep. It's probably the bug in the buffer cache where the kernel would allow userland to queue up so many writes that eventually the kernel is starved out of buffers. Everything else (for example, read operations on your graph files) then sleeps until enough writes have been spilled out to disk. Is there anything (some system tuning?) I can do to get rid of the freezes, or at least to mitigate them? The best solution is an upgrade to -current where this has been fixed. See http://marc.info/?l=openbsd-cvsm=135231065926430w=2 and other related commits by Bob Beck. If you'd rather stick to 5.2 you can try turning off softdep. softdep delays some write operations so turning it off might help somewhat by allowing more read operations to interleave with write operations. While the bug was affecting -current I found that my systems where much more responsive with softdep turned off.
Re: Disk accesses freeze for a lot of seconds
Maybe the following will help. See Tuning for More http://wiki.squid-cache.org/BestOsForSquid I use mount options: noatime and async. I don't use softdep for squid cache either. I found aufs worked best for storage scheme (in squid.conf). I am curious. Anyone out there using diskd? On Sun, Jan 06, 2013 at 07:49:27PM +0100, Lo?c BLOT wrote: I got same problem with squid when squid exit normally (/etc/rc.d/squid stop), when mass squid disk cache is written, there is a one min freeze on the server. (OpenBSD 5.2). The problem was also here under OpenBSD 5.1. CPU is also OK (10% of a big xeon quad). But for me softdeps aren't activated. The temporary solution i used, kill -9 squid process when stop/restart is done. -- Cordialement, Lo??c BLOT, UNIX systems, security and network expert http://www.unix-experience.fr Le dimanche 06 janvier 2013 ?? 16:08 +0100, Federico Giannici a ??crit : You was right: turning off softdep made the freezes much shorter. Thanks. On 01/06/13 13:49, Stefan Sperling wrote: On Sun, Jan 06, 2013 at 12:22:44PM +0100, Federico Giannici wrote: We have an OpenBSD 5.2 amd64 where every 5 minutes a few thousand of .rrd files from MRTG are written (actually, updated) to disk. The problem is that for a few seconds (15-20) every other access to the disk is totally blocked. So during those 15-20 seconds the access to the graphs is freezed! And this is really annoying for a graphs server... It's not a problem of CPU load (it's a quadruple core AMD Athlon II X4 630 Processor). Processes run smoothly, they freeze only when they try to access the disk. Disk is a normal SATA, and the partition is FFS with softdep. It's probably the bug in the buffer cache where the kernel would allow userland to queue up so many writes that eventually the kernel is starved out of buffers. Everything else (for example, read operations on your graph files) then sleeps until enough writes have been spilled out to disk. Is there anything (some system tuning?) I can do to get rid of the freezes, or at least to mitigate them? The best solution is an upgrade to -current where this has been fixed. See http://marc.info/?l=openbsd-cvsm=135231065926430w=2 and other related commits by Bob Beck. If you'd rather stick to 5.2 you can try turning off softdep. softdep delays some write operations so turning it off might help somewhat by allowing more read operations to interleave with write operations. While the bug was affecting -current I found that my systems where much more responsive with softdep turned off.
Re: Disk accesses freeze for a lot of seconds
Maybe the following will help. See Tuning for More http://wiki.squid-cache.org/BestOsForSquid I use mount options: noatime and async. I don't use softdep for squid cache either. that is not good policy. you are asking for trouble.
Re: Disk accesses freeze for a lot of seconds
Maybe the following will help. See Tuning for More http://wiki.squid-cache.org/BestOsForSquid I use mount options: noatime and async. I don't use softdep for squid cache either. that is not good policy. you are asking for trouble. Thanks for the opinion. Yeah I read the disclaimer about async in mount(8) and don't mind taking the risk. As for noatime. Are you kidding me? I forgot tuning = idiot to some on this list. .d.d.
Re: Disk accesses freeze for a lot of seconds
On Mon, Jan 07, 2013 at 12:53:01PM +1000, David Diggles wrote: Maybe the following will help. See Tuning for More http://wiki.squid-cache.org/BestOsForSquid I use mount options: noatime and async. I don't use softdep for squid cache either. that is not good policy. you are asking for trouble. Thanks for the opinion. Yeah I read the disclaimer about async in mount(8) and don't mind taking the risk. As for noatime. Are you kidding me? I briefly looked into adding relatime, but I stopped as soon as I realized there was no more room for additional flags in the existing data structures. I don't have the kernel experience to deal with such headaches properly--that is, without it being excessively ugly or screwing people down the road. Having relatime would be awesome, though.
