Re: immense delayed write to file system (ZFS and UFS2), performance issues
2010/1/18 Morgan Wesström freebsd-questi...@pp.dyndns.biz O. Hartmann wrote: I realise a strange behaviour of several FreeBSD 8.0-STABLE/amd64 boxes. All boxes have the most recent STABLE. One box is a UP system, two others SMP boxes, one with a Q6600 4-core, another XEON with 2x 4-cores (Dell Poweredge III). Symptome: All boxes have ZFS and UFS2 filesystems. Since two weeks or so, sometimes the I/O performance drops massively when doing 'svn update', 'make world' or even 'make kernel'. It doesn't matter what memory and how many cpu the box has, it get stuck for several seconds and freezing. On the UP box, this is sometimes for 10 - 20 seconds. A very interesting phenomenon is the massively delayed file writing on ZFS filesystems I realise. Editing a file in 'vi' running on one XTerm and having in another Xterminal my shell for compiling this file, it takes sometimes up to 20 seconds to get the file updated after it has been written. It's like having an old, slow NFS connection with long cache delays. These massively delayed file transactions are not necessarely under heavy load, sometimes they occur in a relaxed situation. They seem to occur much more often on the UP box than on the SMP boxes, but this strange phenomenon also occur on the Dell Poweredge II, which has 16GB RAM and summa summarum 16 cores. This phenomenon does occur on ZFS- and UFS2 filesystems as well. It is hardly reproducable. Is there any known issue? Ragrds, Oliver The disks involved don't happen to be Western Digital Green Power disks, do they? The Intelli-Park function in these disks are wrecking havoc with I/O in Linux-land at least, causing massive stalls and iowait through the roof during the 25-30 seconds it takes for the heads to unload after parking. I have two of these disks sitting on my desk now collecting dust... /Morgan ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org ZFS is copy on write, therefore to optimize the write performance it delays writes for a long as possible, upto a set maximum time. It will then flush to the disks. How long this time is depends on how much free ram you have available. Assuming processes are eating up all your ram I would imagine you are hitting the max limit. I'm not sure exactly what its set to on bsd but I know the default on opensolaris is 30s. I think this explains your delayed writes. Not sure what will cause the lock ups though. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: immense delayed write to file system (ZFS and UFS2), performance issues
Emil Mikulic wrote: (off-list) On Tue, Jan 19, 2010 at 04:32:34PM +0100, Morgan Wesstr?m wrote: Emil Mikulic wrote: On Tue, Jan 19, 2010 at 09:16:41AM +0100, Gerrit K?hn wrote: Thanks for bringing up this topic here. I have drives showing up close to 80 load cycle counts here. Guess it's time for that fix... :-| Device Model: WDC WD10EACS-00ZJB0 Firmware Version: 01.01B01 Serial Number:WD-WCAS [...] 9 Power_On_Hours 17046 193 Load_Cycle_Count 1045512 The above drive is in a raidz of three. The other two drives from that batch have already failed. :( Did you RMA the failing drives? Did WD comment the Load_Cycle_Count? No. But apparently they're still under warranty, so maybe I'll give that a shot and let you know in ~3 weeks or so. :) Thanks. I'd be really interested to know whether they'll make any comments regarding that value or if they simply ignore it. /Morgan ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: immense delayed write to file system (ZFS and UFS2), performance issues
On 01/18/10 21:34, � wrote: O. Hartmann wrote: I realise a strange behaviour of several FreeBSD 8.0-STABLE/amd64 boxes. All boxes have the most recent STABLE. One box is a UP system, two others SMP boxes, one with a Q6600 4-core, another XEON with 2x 4-cores (Dell Poweredge III). Symptome: All boxes have ZFS and UFS2 filesystems. Since two weeks or so, sometimes the I/O performance drops massively when doing 'svn update', 'make world' or even 'make kernel'. It doesn't matter what memory and how many cpu the box has, it get stuck for several seconds and freezing. On the UP box, this is sometimes for 10 - 20 seconds. A very interesting phenomenon is the massively delayed file writing on ZFS filesystems I realise. Editing a file in 'vi' running on one XTerm and having in another Xterminal my shell for compiling this file, it takes sometimes up to 20 seconds to get the file updated after it has been written. It's like having an old, slow NFS connection with long cache delays. These massively delayed file transactions are not necessarely under heavy load, sometimes they occur in a relaxed situation. They seem to occur much more often on the UP box than on the SMP boxes, but this strange phenomenon also occur on the Dell Poweredge II, which has 16GB RAM and summa summarum 16 cores. This phenomenon does occur on ZFS- and UFS2 filesystems as well. It is hardly reproducable. Is there any known issue? Ragrds, Oliver The disks involved don't happen to be Western Digital Green Power disks, do they? The Intelli-Park function in these disks are wrecking havoc with I/O in Linux-land at least, causing massive stalls and iowait through the roof during the 25-30 seconds it takes for the heads to unload after parking. I have two of these disks sitting on my desk now collecting dust... /Morgan ___ freebsd-sta...@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org The disks in question are indeed WD on one box, but they are all Caviar Black and they performed well months ago with the very same hardware and an earlier FreeBSD 8 version. The other boxes in questions do have a set of mixed type, Seagate, WD, Samsung (mostly Samsung F1 types). We do not use 'Green' drives, due to every box acts as a server and we found green-disks, even from WD, too slow. Oliver ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: immense delayed write to file system (ZFS and UFS2), performance issues
On 01/19/10 10:09, krad wrote: 2010/1/18 Morgan Wesstr�mfreebsd-questi...@pp.dyndns.biz O. Hartmann wrote: I realise a strange behaviour of several FreeBSD 8.0-STABLE/amd64 boxes. All boxes have the most recent STABLE. One box is a UP system, two others SMP boxes, one with a Q6600 4-core, another XEON with 2x 4-cores (Dell Poweredge III). Symptome: All boxes have ZFS and UFS2 filesystems. Since two weeks or so, sometimes the I/O performance drops massively when doing 'svn update', 'make world' or even 'make kernel'. It doesn't matter what memory and how many cpu the box has, it get stuck for several seconds and freezing. On the UP box, this is sometimes for 10 - 20 seconds. A very interesting phenomenon is the massively delayed file writing on ZFS filesystems I realise. Editing a file in 'vi' running on one XTerm and having in another Xterminal my shell for compiling this file, it takes sometimes up to 20 seconds to get the file updated after it has been written. It's like having an old, slow NFS connection with long cache delays. These massively delayed file transactions are not necessarely under heavy load, sometimes they occur in a relaxed situation. They seem to occur much more often on the UP box than on the SMP boxes, but this strange phenomenon also occur on the Dell Poweredge II, which has 16GB RAM and summa summarum 16 cores. This phenomenon does occur on ZFS- and UFS2 filesystems as well. It is hardly reproducable. Is there any known issue? Ragrds, Oliver The disks involved don't happen to be Western Digital Green Power disks, do they? The Intelli-Park function in these disks are wrecking havoc with I/O in Linux-land at least, causing massive stalls and iowait through the roof during the 25-30 seconds it takes for the heads to unload after parking. I have two of these disks sitting on my desk now collecting dust... /Morgan ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org ZFS is copy on write, therefore to optimize the write performance it delays writes for a long as possible, upto a set maximum time. It will then flush to the disks. How long this time is depends on how much free ram you have available. Assuming processes are eating up all your ram I would imagine you are hitting the max limit. I'm not sure exactly what its set to on bsd but I know the default on opensolaris is 30s. I think this explains your delayed writes. Not sure what will cause the lock ups though. ___ freebsd-sta...@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org This could end in a bad situation, where one process writes a files, say with some arbitrary stuff and another successing process is intended to read this file. even if the processes are run serial, those 'delays' could break the chain! The delay situation in a development environment is harsh, but in other circumstances it could develop very bad. I see this strange behaviour now for several weeks, something essential has changed in the code, I guess. On UP boxes the situation is worse sometimes, on SMp boxes with lots of RAM ( 8 and 16 GB and 4 or 8 CPU cores) it is still bad. I have a server that acts as a 'rsync' backup system gathering data from satellite servers from time to time. Since this problem of slowness occured, this 4-core 8 gig RAM box crawls for minutes. Even when X11 is disabled working on console is 'bumpy': terminal out slows down, mouse pointer jumps etc.As I wrote, the same on a 8 core/16 gig box, but not that harsh. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
immense delayed write to file system (ZFS and UFS2), performance issues
I realise a strange behaviour of several FreeBSD 8.0-STABLE/amd64 boxes. All boxes have the most recent STABLE. One box is a UP system, two others SMP boxes, one with a Q6600 4-core, another XEON with 2x 4-cores (Dell Poweredge III). Symptome: All boxes have ZFS and UFS2 filesystems. Since two weeks or so, sometimes the I/O performance drops massively when doing 'svn update', 'make world' or even 'make kernel'. It doesn't matter what memory and how many cpu the box has, it get stuck for several seconds and freezing. On the UP box, this is sometimes for 10 - 20 seconds. A very interesting phenomenon is the massively delayed file writing on ZFS filesystems I realise. Editing a file in 'vi' running on one XTerm and having in another Xterminal my shell for compiling this file, it takes sometimes up to 20 seconds to get the file updated after it has been written. It's like having an old, slow NFS connection with long cache delays. These massively delayed file transactions are not necessarely under heavy load, sometimes they occur in a relaxed situation. They seem to occur much more often on the UP box than on the SMP boxes, but this strange phenomenon also occur on the Dell Poweredge II, which has 16GB RAM and summa summarum 16 cores. This phenomenon does occur on ZFS- and UFS2 filesystems as well. It is hardly reproducable. Is there any known issue? Ragrds, Oliver ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: immense delayed write to file system (ZFS and UFS2), performance issues
O. Hartmann wrote: I realise a strange behaviour of several FreeBSD 8.0-STABLE/amd64 boxes. All boxes have the most recent STABLE. One box is a UP system, two others SMP boxes, one with a Q6600 4-core, another XEON with 2x 4-cores (Dell Poweredge III). Symptome: All boxes have ZFS and UFS2 filesystems. Since two weeks or so, sometimes the I/O performance drops massively when doing 'svn update', 'make world' or even 'make kernel'. It doesn't matter what memory and how many cpu the box has, it get stuck for several seconds and freezing. On the UP box, this is sometimes for 10 - 20 seconds. A very interesting phenomenon is the massively delayed file writing on ZFS filesystems I realise. Editing a file in 'vi' running on one XTerm and having in another Xterminal my shell for compiling this file, it takes sometimes up to 20 seconds to get the file updated after it has been written. It's like having an old, slow NFS connection with long cache delays. These massively delayed file transactions are not necessarely under heavy load, sometimes they occur in a relaxed situation. They seem to occur much more often on the UP box than on the SMP boxes, but this strange phenomenon also occur on the Dell Poweredge II, which has 16GB RAM and summa summarum 16 cores. This phenomenon does occur on ZFS- and UFS2 filesystems as well. It is hardly reproducable. Is there any known issue? Ragrds, Oliver The disks involved don't happen to be Western Digital Green Power disks, do they? The Intelli-Park function in these disks are wrecking havoc with I/O in Linux-land at least, causing massive stalls and iowait through the roof during the 25-30 seconds it takes for the heads to unload after parking. I have two of these disks sitting on my desk now collecting dust... /Morgan ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: immense delayed write to file system (ZFS and UFS2), performance issues
On Tue, 19 Jan 2010, Morgan Wesström wrote: The disks involved don't happen to be Western Digital Green Power disks, do they? The Intelli-Park function in these disks are wrecking havoc with I/O in Linux-land at least, causing massive stalls and iowait through the roof during the 25-30 seconds it takes for the heads to unload after parking. I have two of these disks sitting on my desk now collecting dust... There's this.. http://www.silentpcreview.com/Terabyte_Drive_Fix and you can get the tool at.. http://home.arcor.de/ghostadmin/wdidle3_1_00.zip I am planning to try this out tonight.. -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au The nice thing about standards is that there are so many of them to choose from. -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C signature.asc Description: This is a digitally signed message part.
