Re: Disk I/O as a bottleneck? [OT]
On Sunday, 8 בMay 2011 19:42:55 is...@zahav.net.il wrote: On Sun, 08 May 2011 19:19:25 +0300 guy keren c...@actcom.co.il wrote: and how is all this related to solaris Vs. linux? solaris is *nix, at least was the last time i heard ;) Yes, you are right, but for some reason Solaris has the reputation for handling multicore better than Linux and BSD. Repeating this claim does not make it true: http://www.nas.nasa.gov/Resources/Systems/columbia.html - Smallest node have only 512 (Itanium) cores - Biggest system have 2048 (Itanium) cores - These systems are from ~2006 (5 years ago) - They are SSI (single system image), not clusters. If Solaris was so much better in multicore handling, I'll bet you would be able to point to similar Solaris systems, now can you? -- Oron Peled Voice: +972-4-8228492 o...@actcom.co.il http://users.actcom.co.il/~oron There are two major products that come from Berkeley: LSD and UNIX. We don't believe this to be a coincidence. -- Jeremy S. Anderson ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck? [OT]
On Tue, 10 May 2011 21:57:53 +0300 Oron Peled o...@actcom.co.il wrote: On Sunday, 8 בMay 2011 19:42:55 is...@zahav.net.il wrote: On Sun, 08 May 2011 19:19:25 +0300 guy keren c...@actcom.co.il wrote: and how is all this related to solaris Vs. linux? solaris is *nix, at least was the last time i heard ;) Yes, you are right, but for some reason Solaris has the reputation for handling multicore better than Linux and BSD. Repeating this claim does not make it true: Reading what I wrote might help you a little. I said Solaris has the *reputation* for handling multicore better. http://www.nas.nasa.gov/Resources/Systems/columbia.html - Smallest node have only 512 (Itanium) cores - Biggest system have 2048 (Itanium) cores - These systems are from ~2006 (5 years ago) - They are SSI (single system image), not clusters. If Solaris was so much better in multicore handling, I'll bet you would be able to point to similar Solaris systems, now can you? I'm sure I could if I was an Oracle marketing rep, but I really don't care. I use Linux, Solaris, and 3 different BSD and they all have their good and bad points. YMMV. I didn't bother to look at your link but if you don't like what I wrote, think of the value of comparing whatever OS you are talking about running on *Itanium* to running it on x86. That's not exactly a sound basis for comparison :rolleyes: I was suggesting to the OP it might be good to try a different OS on *his* hardware if he has a nice box and isn't happy with the performance. I don't see the crime in that and sorry some of you found it so offensive. Maybe you should just buy him an Itanium farm if you feel so strongly about it ;-) Have a nice day. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Mon, 2011-05-09 at 06:29 +0300, Omer Zak wrote: [This E-mail message is bottom-posting contrary to my usual custom.] Thanks :) 1. Kernel version? 2.6.38 added a very small patch that that done wonders to eliminate foreground process scheduling issues that plagued desktop setups since the introduction of the CFS scheduler*. (Google for +2.6.38 +group +scheduling) On my Fedora 14 + 2.6.38 / dual 6C Xeon workstation I can easily watching a DVD movies while compiling the kernel and running a couple of VM's (using qemu-kvm) in the background. Doing the same using the stock Fedora 14 2.6.35 kernel is... err... interesting experience :) Standard Debian Squeeze kernel: $ uname -a Linux c4 2.6.32-5-vserver-amd64 #1 SMP Mon Mar 7 23:14:47 UTC 2011 x86_64 GNU/Linux I'd consider trying a kernel from debian-testing or rolling your own. As I said, at least in my experience, the interface feel of 2.6.38 is -far-, -far- better than previous kernels. 2. Are you sure your SATA is in AHCI mode? (Simply type: $ find /sys/devices | grep ahci) My SATA indeed runs in non-AHCI mode. The question now is how to configure it to run in AHCI mode? According to the Wikipedia article http://en.wikipedia.org/wiki/Advanced_Host_Controller_Interface, I need to modify a BIOS setting and maybe create a new initrd image. Another source: http://www.techmetica.com/howto/sata-ahci-mode-bios-setting.-what-does-it-do/ I can handle the BIOS setting but how to check whether a new initrd image is needed, and if necessary create it? First you'll need to enable it in your BIOS. (Usually under SATA controller mode, under advanced settings) If you cannot seem to find this option, trying searching for a BIOS upgrade for your motherboard. If it's enabled, and somehow your kernel haven't picked it up, check that CONFIG_SATA_AHCI is enabled in your kernel .confing. ($ cat /boot/*config* | grep AHCI) ... If it does, you may need an updated kernel that better supports your SATA chipset. (Which board is it?) - Gilboa ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
How to check initrd's contents? (was: Re: Disk I/O as a bottleneck?)
Hello Gilboa, On Mon, 2011-05-09 at 09:43 +0300, Gilboa Davara wrote: 1. Kernel version? Standard Debian Squeeze kernel: $ uname -a Linux c4 2.6.32-5-vserver-amd64 #1 SMP Mon Mar 7 23:14:47 UTC 2011 x86_64 GNU/Linux I'd consider trying a kernel from debian-testing or rolling your own. As I said, at least in my experience, the interface feel of 2.6.38 is -far-, -far- better than previous kernels. The shell feel is OK - when I want to open another shell, one opens promptly (does not need any disk I/O as the relevant executables are still cached in memory thanks to already-open shells; and there are idle cores). So I am not in hurry to install a more recent kernel. 2. Are you sure your SATA is in AHCI mode? (Simply type: $ find /sys/devices | grep ahci) My SATA indeed runs in non-AHCI mode. The question now is how to configure it to run in AHCI mode? According to the Wikipedia article http://en.wikipedia.org/wiki/Advanced_Host_Controller_Interface, I need to modify a BIOS setting and maybe create a new initrd image. Another source: http://www.techmetica.com/howto/sata-ahci-mode-bios-setting.-what-does-it-do/ I can handle the BIOS setting but how to check whether a new initrd image is needed, and if necessary create it? First you'll need to enable it in your BIOS. (Usually under SATA controller mode, under advanced settings) If you cannot seem to find this option, trying searching for a BIOS upgrade for your motherboard. If it's enabled, and somehow your kernel haven't picked it up, check that CONFIG_SATA_AHCI is enabled in your kernel .confing. ($ cat /boot/*config* | grep AHCI) ... If it does, you may need an updated kernel that better supports your SATA chipset. (Which board is it?) My kernel is configured to have AHCI as a module: CONFIG_SATA_AHCI=m However I understand that it means that this module is needed also in the initrd image. How can I check which modules made it to the initrd image last time it was rebuilt? Thanks, --- Omer -- If verbal consent is not obtained in triplicate, it is date rape. Asking permission constitutes harassment. My opinions, as expressed in this E-mail message, are mine alone. They do not represent the official policy of any organization with which I may be affiliated in any way. WARNING TO SPAMMERS: at http://www.zak.co.il/spamwarning.htmlDelay is the deadliest form of denial.C. Northcote Parkinson My own blog is at http://www.zak.co.il/tddpirate/ My opinions, as expressed in this E-mail message, are mine alone. They do not represent the official policy of any organization with which I may be affiliated in any way. WARNING TO SPAMMERS: at http://www.zak.co.il/spamwarning.html ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: How to check initrd's contents? (was: Re: Disk I/O as a bottleneck?)
On Mon, May 09, 2011 at 03:18:08PM +0300, Omer Zak wrote: My kernel is configured to have AHCI as a module: CONFIG_SATA_AHCI=m However I understand that it means that this module is needed also in the initrd image. How can I check which modules made it to the initrd image last time it was rebuilt? something like mkdir /some/temp/dir cd /some/temp/dir gzip -dc /boot/my-initrd | cpio -idm That is, assuming it's a ramfs-based initrd and not an old-style filesystem image. -- Didi ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: How to check initrd's contents? (was: Re: Disk I/O as a bottleneck?)
On Mon, 2011-05-09 at 15:30 +0300, Yedidyah Bar-David wrote: On Mon, May 09, 2011 at 03:18:08PM +0300, Omer Zak wrote: My kernel is configured to have AHCI as a module: CONFIG_SATA_AHCI=m However I understand that it means that this module is needed also in the initrd image. How can I check which modules made it to the initrd image last time it was rebuilt? something like mkdir /some/temp/dir cd /some/temp/dir gzip -dc /boot/my-initrd | cpio -idm That is, assuming it's a ramfs-based initrd and not an old-style filesystem image. Thanks, my initrd images do include the module /lib/modules/kernel-version/kernel/drivers/ata/ahci.ko So all I need to do is to modify the appropriate BIOS setup setting - will wait until next reboot opportunity! --- Omer -- Eli Marmor's Law: netiquette forbids people from replying I agree with you to messages in mailing lists, skewing discussions in favor of the troublemaker type of dissenters. My own blog is at http://www.zak.co.il/tddpirate/ My opinions, as expressed in this E-mail message, are mine alone. They do not represent the official policy of any organization with which I may be affiliated in any way. WARNING TO SPAMMERS: at http://www.zak.co.il/spamwarning.html ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: How to check initrd's contents? (was: Re: Disk I/O as a bottleneck?)
On Mon, 2011-05-09 at 15:49 +0300, Omer Zak wrote: On Mon, 2011-05-09 at 15:30 +0300, Yedidyah Bar-David wrote: On Mon, May 09, 2011 at 03:18:08PM +0300, Omer Zak wrote: My kernel is configured to have AHCI as a module: CONFIG_SATA_AHCI=m However I understand that it means that this module is needed also in the initrd image. How can I check which modules made it to the initrd image last time it was rebuilt? something like mkdir /some/temp/dir cd /some/temp/dir gzip -dc /boot/my-initrd | cpio -idm That is, assuming it's a ramfs-based initrd and not an old-style filesystem image. Thanks, my initrd images do include the module /lib/modules/kernel-version/kernel/drivers/ata/ahci.ko So all I need to do is to modify the appropriate BIOS setup setting - will wait until next reboot opportunity! --- Omer make sure that the 'init' file in the initrd indeed loads this module (it's a shell script) before mounting the root file-system from the hard disk. --guy ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, May 08, 2011 at 07:28:49AM +0300, Nadav Har'El wrote: On Sat, May 07, 2011, guy keren wrote about Re: Disk I/O as a bottleneck?: and if you have a lot of money to spend - you could consider buying an enterprise-grade SSD (e.g. from fusion I/O or from OCZ - although for your use-case, some of the cheaper SSDs will do) and use it instead of the hard disks. they only cost thousands of dollars for a 600GB SSD ;) Instead of buying a huge SSD for thousands of dollars another option you might consider is to buy a relatively small SSD with just enough space to hold your / partition and swap space. Even 20 G may be enough. The rest of your disk - holding your source code, photos, songs, movies, or whatever you typically fill a terabyte with, will be a normal, cheap, hard disk. Several of my friends have gone with such a setup on their latest computer, and they are very pleased. I am considering, for my next laptop, and taking into account the fact that most laptops do not have space for two disks but do have some kind of flash memory slot (card reader) - usually SD-something, to have the OS on a (e.g.) SD card of 16 or 32 GB. I have no other experience with such cards, so I do not know if they are considered durable enough, fast enough - both random and sequential IO, both compared to SATA mechanical disks and to SATA flash ones, etc. Comments are welcome :-) -- Didi ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sat, May 07, 2011, Omer Zak wrote about Re: Disk I/O as a bottleneck?: I suspect that speeding up /usr won't help improve performance that much. The applications, which seem to be sluggish, deal with a lot of user data in /home. Furthermore, this user data varies a lot with time, hence it is not that good idea to store it in SSD. As usual, it depends on your workload applies. In my own personal experience (and naturally, it might differ considerably from your use case), when I see sluggish behavior on a desktop machine, what is actually happening is that foreground activity, such as playing or working with video files, or such as compilation of a large project, causes a lot of other pages to be swapped out; And then, when you switch to a different application, it needs to swap pages in - either program text (code) directly from the executables, or data pages from the swap partition. So when you switch to a GUI application, suddenly it takes a second to respond to a mouse click (it needs to swap in the relevant code and data), and when you type ls in a shell it takes much longer than usual (both the ls code and the directory are not in memory). Not only does fetching all these missing pages require a lot of seeks, which are slow on hard disks, it's even worse when that other application (which is using all that user data), continues to do a lot of seeks, and competes with the seeks needed to fetch the missing pages. In such a case if your system files - binaries, shared libraries, and swap, would be on a separate disk, everything might feel more responsive. If that separate disk had low seek times and hight throughput, it would be especially quick to recover from swap-outs, so you might see even better interactive behavior. Like I said, several of my friends tried this setup (SSD+hard disk) and liked the improved feel of the system (and the faster boot :-)). I haven't tried it myself, though. -- Nadav Har'El|Sunday, May 8 2011, 4 Iyyar 5771 n...@math.technion.ac.il |- Phone +972-523-790466, ICQ 13349191 |Christopher Robin Hood steals from the http://nadav.harel.org.il |rich and gives to the Pooh. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, May 08, 2011, is...@zahav.net.il wrote about Re: Disk I/O as a bottleneck?: I don't agree with this setup. Regular consumer drives setup with RAID to stripe are going to be much, much faster and have less problems in the long run than single SSDs at this point as well as being a better value until prices change a lot. Having two hard disks will, at best case, *double* your seek time. This is still pretty slow, isn't it? Won't an SSD, even cheap one, have a better random access read performance? Consider not using swap, because swap when in use causes a lot of thrashing and kills performance especially if you only have 1 or 2 drives. If you have Even without any swap space, you can have a lot of thrashing: Clean pages - program text (the code), shared libraries, memory-mapped files, and so on - are simply forgotten when the page cache is needed for other things, and when you get back to that other program, suddenly all its code and libraries are not in memory, and getting them back requires a lot of random-access reads from disk, and (especially if the disk is doing other things at the same time) causes the program to appear stuck or sluggish. Again, I don't know if this is the sort of problem that actually bothered the OP. -- Nadav Har'El|Sunday, May 8 2011, 4 Iyyar 5771 n...@math.technion.ac.il |- Phone +972-523-790466, ICQ 13349191 |The only intuitive interface is the http://nadav.harel.org.il |nipple. After that, it's all learned. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, May 8, 2011 at 7:28 AM, Nadav Har'El n...@math.technion.ac.ilwrote: Instead of buying a huge SSD for thousands of dollars another option you might consider is to buy a relatively small SSD with just enough space to hold your / partition and swap space. Even 20 G may be enough. The rest of your disk - holding your source code, photos, songs, movies, or whatever you typically fill a terabyte with, will be a normal, cheap, hard disk. Several of my friends have gone with such a setup on their latest computer, and they are very pleased. I have set up my latest system just like that. Though mine was a bit pricey: I went for the Intel X25-E 32GB. The OS and homedir are on it; Large datasets go on various Samsung SpinPoint 1TB F3 drives I've installed as well. The system is already more than a year old, and the free space is 20%, which, I am assuming, means I've already filled the disk (due to deletes and the SSD wear-leveling algorithms) and already doing erases, andthe performance is still nothing short of AMAZING - sub-1ms seek time is a great thing when you scan the filesystem etc. It just feels as if Disk I/O is no longer my bottleneck (and the CPU is a Quad Core AMD PhenomII 955 with 8GB RAM...). Of course - I don't use swap. Performance after 1 year as mentioned: # hdparm -t /dev/sda /dev/sda: Timing buffered disk reads: 722 MB in 3.00 seconds = 240.27 MB/sec As always, YMMV :) -- Shimi ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On May 8, 2011, at 9:30 AM, Yedidyah Bar-David wrote: I am considering, for my next laptop, and taking into account the fact that most laptops do not have space for two disks but do have some kind of flash memory slot (card reader) - usually SD-something, to have the OS on a (e.g.) SD card of 16 or 32 GB. I have no other experience with such cards, so I do not know if they are considered durable enough, fast enough - both random and sequential IO, both compared to SATA mechanical disks and to SATA flash ones, etc. Comments are welcome :-) It depends upon how you do it. The main difference in this case between a SOLID STATE DISK and memory card is the number of times you can write on it before it stops working. Modern memory cards do not use the same physical location for data all the time. The card itself randomizes where you write data, so that the useage of each bit on the card is spread out evenly. Of course this only works if the card is not full, and the emptier it is the better off you are. Whether this works with *NIX file systems is another question and I can't answer it. One of the bad things is that standard *NIX files systems are designed with magnetic media in mind, they update the access time of files every time you open them. This is bad for files that are opened often. The way around this is to mount a file system read only. Using a compressed read only file system, such as that on a live CD works well in this case. The problem with it is that you can't add software or change settings. UBUNTU has a setup where you can install a live system to a memory card/stick and it will mount your home directory in the unused space. If you can live with the limitations, then it will work for you. I think someone else said to use a small SSD for the system and a hard disk for your data. This would work extremely well for this situation where instead of a hard disk, you used a memory stick or card for it. It also depends upon what you are doing with it. Besides entertainment, my needs are fullfilled with an Xterm type terminal, SSH, a web browser and an email program. For entertainment, an MP3 player and one that will play 360P videos is enough. This can be accomplised with a lower power processor (Intel Atom for example) and a small screen. While you can get laptops with 15 inch screens and I7 processors, I'm not sure you would gain anything except a higher price by replacing a disk with an SSD/memory card combo. The latest Apple rumor is that they are going to produce a laptop soon with an ARM processor.Based on the success of the iPad, it probably will be a netbook size screen, a multicore ARM processor and a keyboard. It may or may not have a touch screen. I'm hoping that this rumor, whether there is any truth to it or not will fuel development of small ARM based netbooks. Unfortunately netbooks instead of getting smaller and cheaper, have gone the other way and become more expensive, larger, heavier and more powerfull. Geoff. -- Geoffrey S. Mendelson, N3OWJ/4X1GM Those who cannot remember the past are condemned to misquote it. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, May 8, 2011 at 10:02 AM, geoffrey mendelson geoffreymendel...@gmail.com wrote: One of the bad things is that standard *NIX files systems are designed with magnetic media in mind, they update the access time of files every time you open them. This is bad for files that are opened often. man mount search for 'noatime' -- Shimi ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, May 08, 2011, geoffrey mendelson wrote about Re: Disk I/O as a bottleneck?: One of the bad things is that standard *NIX files systems are designed with magnetic media in mind, they update the access time of files every time you open them. This is bad for files that are opened often. The way around this is to mount a file system read only. Using a Instead of mounting with the ro (read-only) option, a better option to use would be noatime, which disables just the updating of access-time. See also the nodiratime, relatime. These options are explained in mount(8). -- Nadav Har'El|Sunday, May 8 2011, 4 Iyyar 5771 n...@math.technion.ac.il |- Phone +972-523-790466, ICQ 13349191 |Happiness isn't getting what you want, http://nadav.harel.org.il |it's wanting what you've got. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, May 08, 2011, Nadav Har'El wrote about Re: Disk I/O as a bottleneck?: Having two hard disks will, at best case, *double* your seek time. This is Of course, I meant *half*, not *double* :-) -- Nadav Har'El|Sunday, May 8 2011, 4 Iyyar 5771 n...@math.technion.ac.il |- Phone +972-523-790466, ICQ 13349191 |I have an open mind - it's just closed http://nadav.harel.org.il |for repairs. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, 08 May 2011 09:55:35 +0300 Nadav Har'El n...@math.technion.ac.il wrote: On Sun, May 08, 2011, is...@zahav.net.il wrote about Re: Disk I/O as a bottleneck?: I don't agree with this setup. Regular consumer drives setup with RAID to stripe are going to be much, much faster and have less problems in the long run than single SSDs at this point as well as being a better value until prices change a lot. Having two hard disks will, at best case, *double* your seek time. This is still pretty slow, isn't it? Won't an SSD, even cheap one, have a better random access read performance? Striping works because the controllers can overlap I/O. There's no reason you have to seek on two drives, that's the *worst* case and probably shouldn't happen. The best case is the controller knows the data is on one drive and seek remains the same. Of course a physical drive is always more beneficial for sequential access than random, but striping is still better for random than not striping because of the caching- your system may be doing other work that needs data found on the other drive, and that means seek that you totally avoid. The more drives you spread your data across the less likely you are to thrash any one drive. As we say, six barbers, no waiting. I don't think any cheap SSDs cache enough to beat a good striped setup of even 2 physical disks and the page sizes aren't usually matched to Linux's requirements. I read an article about how SSD design was mostly made to Windows filesystems and can work against you somewhat if you use them on Linux or other OS. I don't know how true it was but it makes you think. Anyway as you add more disks you keep increasing performance with a striping setup, there is no downside except if you mirror but then again there is no free lunch. You can have it fast or reliable or cheap, but you can't have it all. The question is how much RAID and how many drives you can buy for the price of a good SSD, how much cache will you get with striped RAID and how much cache do you get with a good SSD, and how long will everything last? SSD still has some nasty issues like wear leveling, lifetime problems, etc. Enterprise servers are still using physical drives. Those guys can pay anything and they buy the old stuff so that must tell us something. Personally I'm staying with drives that spin until technology improves and I start reading about production servers that are totally SSD. YMMV ;-) Consider not using swap, because swap when in use causes a lot of thrashing and kills performance especially if you only have 1 or 2 drives. If you have Even without any swap space, you can have a lot of thrashing: Clean pages - program text (the code), shared libraries, memory-mapped files, and so on - are simply forgotten when the page cache is needed for other things, and when you get back to that other program, suddenly all its code and libraries are not in memory, and getting them back requires a lot of random-access reads from disk, and (especially if the disk is doing other things at the same time) causes the program to appear stuck or sluggish. That's true but swap only makes it worse since it's many magnitudes slower than RAM and in my experience, Linux doesn't do a very good job of managing it. If I have 4G of RAM and less than 2G is in use, I still see a few meg being used by swap. What's the reason for that? I don't want to start swapping until my RAM is very close to maxxed out. And then I want swap to get cleaned out very quickly after RAM is available. Swapping should be an absolute last resort on a modern machine. It looks like swap processing is something that hasn't received much attention since it was designed and things have changed a lot and it's time to stop using it until they revisit how it should work. If you have a monitoring program that tells you how much RAM and swap are in use, you will see a dramatic slowdown in performance when you swap even a little. I think the OP said he had a fast CPU and tons of RAM so if the system feels sluggish then the obvious things to look at are filesystem layout, choice of filesystems, turning off swap. If that doesn't work then maybe look into another OS...at this point Linux (and BSD) still aren't doing SMP as well as other OS but there may not be any better choices on Intel hardware. As you or somebody else pointed out, many apps don't thread enough to exploit a hyperthreading quad. Maybe the OP should look into running Solaris, which is known to do well in this area. It's probably going to be a long time until Linux and Linux apps really exploit multicore well. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, 08 May 2011 10:02:18 +0300 geoffrey mendelson geoffreymendel...@gmail.com wrote: One of the bad things is that standard *NIX files systems are designed with magnetic media in mind, they update the access time of files every time you open them. This is bad for files that are opened often. You can stop this with noatime in the fstab. It's something I do whenever I set up a new install. The way around this is to mount a file system read only. Using a compressed read only file system, such as that on a live CD works well in this case. The problem with it is that you can't add software or change settings. You don't need to mount the fs r/o to use noatime. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, 08 May 2011 10:15:30 +0300 Nadav Har'El n...@math.technion.ac.il wrote: On Sun, May 08, 2011, Nadav Har'El wrote about Re: Disk I/O as a bottleneck?: Having two hard disks will, at best case, *double* your seek time. This is Of course, I meant *half*, not *double* :-) Actually it might do a lot better than that because of caching. Aside from the fact striped data is spread across drives, it exploits caching better since data from more than one MRU file is likely to be in cache. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, 2011-05-08 at 09:57 +0300, shimi wrote: On Sun, May 8, 2011 at 7:28 AM, Nadav Har'El n...@math.technion.ac.il wrote: Instead of buying a huge SSD for thousands of dollars another option you might consider is to buy a relatively small SSD with just enough space to hold your / partition and swap space. Even 20 G may be enough. The rest of your disk - holding your source code, photos, songs, movies, or whatever you typically fill a terabyte with, will be a normal, cheap, hard disk. Several of my friends have gone with such a setup on their latest computer, and they are very pleased. I have set up my latest system just like that. Though mine was a bit pricey: I went for the Intel X25-E 32GB. The OS and homedir are on it; Large datasets go on various Samsung SpinPoint 1TB F3 drives I've installed as well. The system is already more than a year old, and the free space is 20%, which, I am assuming, means I've already filled the disk (due to deletes and the SSD wear-leveling algorithms) and already doing erases, andthe performance is still nothing short of AMAZING - sub-1ms seek time is a great thing when you scan the filesystem etc. It just feels as if Disk I/O is no longer my bottleneck (and the CPU is a Quad Core AMD PhenomII 955 with 8GB RAM...). Of course - I don't use swap. Performance after 1 year as mentioned: # hdparm -t /dev/sda /dev/sda: Timing buffered disk reads: 722 MB in 3.00 seconds = 240.27 MB/sec As always, YMMV :) what tends to get worse after the SSD becomes full is writes, not reads. and combinations of reads and writes make things look worse (the writes slow down the reads). however, if you feel that the system is very fast after one year of use - that's good enough for me. do you have the ability to extract wear leveling information from your SSD? it would be interesting to know whether the drive is being used in a manner that will indeed comply with the life-time expentency it is sold with (5 years?), or better, or worse. --guy ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, 2011-05-08 at 09:30 +0300, Yedidyah Bar-David wrote: On Sun, May 08, 2011 at 07:28:49AM +0300, Nadav Har'El wrote: On Sat, May 07, 2011, guy keren wrote about Re: Disk I/O as a bottleneck?: and if you have a lot of money to spend - you could consider buying an enterprise-grade SSD (e.g. from fusion I/O or from OCZ - although for your use-case, some of the cheaper SSDs will do) and use it instead of the hard disks. they only cost thousands of dollars for a 600GB SSD ;) Instead of buying a huge SSD for thousands of dollars another option you might consider is to buy a relatively small SSD with just enough space to hold your / partition and swap space. Even 20 G may be enough. The rest of your disk - holding your source code, photos, songs, movies, or whatever you typically fill a terabyte with, will be a normal, cheap, hard disk. Several of my friends have gone with such a setup on their latest computer, and they are very pleased. I am considering, for my next laptop, and taking into account the fact that most laptops do not have space for two disks but do have some kind of flash memory slot (card reader) - usually SD-something, to have the OS on a (e.g.) SD card of 16 or 32 GB. I have no other experience with such cards, so I do not know if they are considered durable enough, fast enough - both random and sequential IO, both compared to SATA mechanical disks and to SATA flash ones, etc. Comments are welcome :-) SD cards are much much much slower then SSDs with regards to sequential I/O - and i think they live a shorter life. if you want to use one - you should make sure it's set in a mostly read-only setup. problematic directories include /var and /tmp. specifically, installing RPMs on such drives works much slower then on hard disks. they are still used on various appliances and embedded systems, in such a mostly read-only configuration. --guy ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, May 8, 2011 at 12:01 PM, guy keren c...@actcom.co.il wrote: On Sun, 2011-05-08 at 09:57 +0300, shimi wrote: what tends to get worse after the SSD becomes full is writes, not reads. and combinations of reads and writes make things look worse (the writes slow down the reads). You're of course correct. Hope this satisfies the issue: $ cat test.sh #!/bin/sh dd if=/dev/zero of=test123456.dat bs=10 count=1 sync $ time ./test.sh 1+0 records in 1+0 records out 10 bytes (1.0 GB) copied, 3.89247 s, 257 MB/s real0m6.158s user0m0.001s sys 0m1.738s (obviously dd itself has stuff in RAM. this is why I used time with sync after the dd. 1GB in 6.158 seconds is 162MB/s not too bad. still better than the Samsung F3 which is one of the fastest disks out there... same script on that 1TB drive takes 12.239s to complete the same task..) however, if you feel that the system is very fast after one year of use - that's good enough for me. I do. And I don't think it's such a difference. Most writes are pretty small, and will not halt the system. I think most of the time the system is slow due to the heads busy with moving around the platter (seek), something that is almost completely eliminated in SSD - and *that's* why you have the performance boost. Correct, there are lousy SSDs that write very slowly, and then block I/O to the lengthy erase process, and will hang the app or the whole bus (depends on the controller, I guess?)... but I don't think the X25-E falls under that category :) do you have the ability to extract wear leveling information from your SSD? it would be interesting to know whether the drive is being used in a manner that will indeed comply with the life-time expentency it is sold with (5 years?), or better, or worse. I don't know, how do you extract such information? The rated MTBF of my specific drive is 2 million hours. If I still know my math, that's some 228 years -- Shimi ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On The rated MTBF of my specific drive is 2 million hours. If I still know my math, that's some 228 years Which is meaningless. The life expectency of a drive is closer to the length of the warranty period. Warranties are decided based upon projected return rates. The manufacturers want no more than a 5% return rate, some less such as 2% or 3%. Once they expect more to come back, they no longer provide a warranty. So if they expect that 2% will come back in the first 3 years. They give you a 3 year warranty. These warranties only apply to retail drives. OEM drives generally are sold without a warranty at all. The OEM provides a warranty for the entire system, and negotiates a lower price with the understanding that they will eat any returns in exchange. That's even starting to affect computers, I saw in Friday's Yediot a computer sold for 15% less if you took it with a one year warranty instead of a three year one. Since it was a low end computer, possibly obsolete in a year or two, it may have been worth it. On the other hand my son is chomping at the bit for the one year warranty to expire on his computer so he can talk all of his relatives into chipping in and buying him a new video card. Last year's high end video card from last year is not fast enough now. :-) I on the other hand was recently given a five year old computer, which due to memory restrictions and a lousy BIOS will be permanently stuck on Windows XP. A new hard drive, a fresh install of Linux, and I'm happy. I/O is absymal, but the CPU is fast. Geoff. -- Geoffrey S. Mendelson, N3OWJ/4X1GM Those who cannot remember the past are condemned to misquote it. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, 2011-05-08 at 12:26 +0300, shimi wrote: On Sun, May 8, 2011 at 12:01 PM, guy keren c...@actcom.co.il wrote: On Sun, 2011-05-08 at 09:57 +0300, shimi wrote: what tends to get worse after the SSD becomes full is writes, not reads. and combinations of reads and writes make things look worse (the writes slow down the reads). You're of course correct. Hope this satisfies the issue: $ cat test.sh #!/bin/sh dd if=/dev/zero of=test123456.dat bs=10 count=1 sync $ time ./test.sh 1+0 records in 1+0 records out 10 bytes (1.0 GB) copied, 3.89247 s, 257 MB/s real0m6.158s user0m0.001s sys 0m1.738s (obviously dd itself has stuff in RAM. this is why I used time with sync after the dd. 1GB in 6.158 seconds is 162MB/s not too bad. still better than the Samsung F3 which is one of the fastest disks out there... same script on that 1TB drive takes 12.239s to complete the same task..) however, if you feel that the system is very fast after one year of use - that's good enough for me. I do. And I don't think it's such a difference. Most writes are pretty small, and will not halt the system. I think most of the time the system is slow due to the heads busy with moving around the platter (seek), something that is almost completely eliminated in SSD - and *that's* why you have the performance boost. Correct, there are lousy SSDs that write very slowly, and then block I/O to the lengthy erase process, and will hang the app or the whole bus (depends on the controller, I guess?)... but I don't think the X25-E falls under that category :) do you have the ability to extract wear leveling information from your SSD? it would be interesting to know whether the drive is being used in a manner that will indeed comply with the life-time expentency it is sold with (5 years?), or better, or worse. I don't know, how do you extract such information? The rated MTBF of my specific drive is 2 million hours. If I still know my math, that's some 228 years -- Shimi wear leveling has nothing to do with MTBF. once you write ~100,000 times to a single cell in the SSD - it's dead. due to the wear leveling methods of the SSD - this will happen once you write ~100,000 times to all cell groups on the SSD - assuming the wear-leveling algorithm of the SSD is implemented without glitches. note that these writes don't come only from the host - many of them are generated internally by the SSD, due to its wear-leveling algorithms. an SSD could perform several writes for each host-initiated write operation on average. intel claims their X25-E has very impressive algorithms in this regard. it'll be interesting to check these claims with the actual state of your SSD. fetching wear-leveling info is SSD-dependent. you'll need to check if intel provides a tool to do that on linux, for your SSD. --guy ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, May 8, 2011 at 1:21 PM, guy keren c...@actcom.co.il wrote: On Sun, 2011-05-08 at 12:26 +0300, shimi wrote: On Sun, May 8, 2011 at 12:01 PM, guy keren c...@actcom.co.il wrote: do you have the ability to extract wear leveling information from your SSD? it would be interesting to know whether the drive is being used in a manner that will indeed comply with the life-time expentency it is sold with (5 years?), or better, or worse. I don't know, how do you extract such information? The rated MTBF of my specific drive is 2 million hours. If I still know my math, that's some 228 years -- Shimi wear leveling has nothing to do with MTBF. once you write ~100,000 times to a single cell in the SSD - it's dead. due to the wear leveling methods of the SSD - this will happen once you write ~100,000 times to all cell groups on the SSD - assuming the wear-leveling algorithm of the SSD is implemented without glitches. I know... I was referring more to the life time expectancy it is sold with when I quoted that number. Unless block write fails do not consist of a failure, and if not, I don't know what is, in an SSD :). Obviously on a DB data disk it is going to happen much faster than on my desktop. b.t.w. IIRC when a cell dies, it does so gracefully; I.e. no data is lost, and there are spare blocks for that case... and even when they're all full, you just get to the point that you still have your data read-only. I vaguely remember I read that somewhere... and if it's indeed like that, this is still way better than a regular hard drive - those tend to usually take all your data with them, and are much more sensitive to many things (shock - physical/electric, heat, etc...) note that these writes don't come only from the host - many of them are generated internally by the SSD, due to its wear-leveling algorithms. an SSD could perform several writes for each host-initiated write operation on average. intel claims their X25-E has very impressive algorithms in this regard. it'll be interesting to check these claims with the actual state of your SSD. I know that too :) fetching wear-leveling info is SSD-dependent. you'll need to check if intel provides a tool to do that on linux, for your SSD. Google didn't help me a lot in that regard (I understand there's a Win$ SW maybe to do that; Unfortunately, I don't have code from Redmond anywhere inside my chassis.) So if you find something yourself, I'm willing to test... -- Shimi ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
SSD? (Re: Disk I/O as a bottleneck?)
On Sun, 2011-05-08 at 13:27 +0300, shimi wrote: b.t.w. IIRC when a cell dies, it does so gracefully; I.e. no data is lost, and there are spare blocks for that case... and even when they're all full, you just get to the point that you still have your data read-only. I vaguely remember I read that somewhere... and if it's indeed like that, this is still way better than a regular hard drive - those tend to usually take all your data with them, and are much more sensitive to many things (shock - physical/electric, heat, etc...) While its very unlikely that you'll ever come close to reaching the (more-or-less-theoretical) per-cell write limit, SSD's do have their share of major issues that should be taken into account: 1. Bricking: As a general rule, SSD's are -far- more vulnerable to data corruption and data loss issues and the reason for this is rather simple: Compared to spinning HDD's, SSD's have highly complex firmware that uses complex write-leveling algorithms and, in some cases, complex compression schemes. (SandForce SF-1xxx, SF-2xxx controllers) As a result, any corner bug in the SSD's firmware will usually brick your SSD, leaving you with nothing. Far worse, while we (as an industry) have extensive of experience in yanking data out of a damaged spinning HDD's, there's far less collective experience in dealing with bricked SDD's. 2. Experience: We literally have decades of experience with dealing with spinning HDD's. Give me a RAID with 1,000 drives and I can predict how many drives will die each month, each year. On the other hand, we simply do not have sufficient experience in dealing with high-end SSD's to draw the predict the same when it comes to SSD's. 3. Performance: TRIM, file-system optimization, dealing with drives running at 90% full capacity, etc. Getting the most out of an SSD's requires careful planning. Let me be clear: I do not doubt that SSD's will slowly replace spinning drives in 90% of all the use cases. I am saying that I'd consider twice before using SDD's in mission critical environment on in places with inadequate backup strategy. - Gilboa ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sat, 2011-05-07 at 15:29 +0300, Omer Zak wrote: I have a PC with powerful processor, lots of RAM and SATA hard disk. Nevertheless I noticed that sometimes applications (evolution E-mail software and Firefox[iceweasel] Web browser) have the sluggish feel of a busy system (command line response time remains crisp, however, because the processor is 4x2 core one [4 cores, each multithreads as 2]). I run the gnome-system-monitor all the time. I notice that even when those applications feel sluggish, only one or at most two CPUs have high utilization, and there is plenty of free RAM (no swap space is used at all). Disk I/O is not monitored by gnome-system-monitor. So I suspect that the system is slowed down by disk I/O. I would like to eliminate it as a possible cause for the applications' sluggish feel. I ran smartctl tests on the hard disk, and they gave it clean bill of health. Therefore I/O error recovery should not be the reason for performance degradation. I am asking Collective Wisdom for advice about how to do: 1. Monitoring disk I/O load (counting I/O requests is not sufficient, as each request takes different time to complete due for example to disk head seeks or platter rotation time). 2. Disk scheduler fine-tuning possibilities to optimize disk I/O handling. 3. If smartctl is not sufficient to ensure that no I/O error overhead is incurred, how to better assess the hard disk's health? Thanks, --- Omer Hello Omer, Two questions: 1. Kernel version? 2.6.38 added a very small patch that that done wonders to eliminate foreground process scheduling issues that plagued desktop setups since the introduction of the CFS scheduler*. (Google for +2.6.38 +group +scheduling) On my Fedora 14 + 2.6.38 / dual 6C Xeon workstation I can easily watching a DVD movies while compiling the kernel and running a couple of VM's (using qemu-kvm) in the background. Doing the same using the stock Fedora 14 2.6.35 kernel is... err... interesting experience :) 2. Are you sure your SATA is in AHCI mode? (Simply type: $ find /sys/devices | grep ahci) - Gilboa * At least in my case... ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, 2011-05-08 at 07:31 +, is...@zahav.net.il wrote: at this point Linux (and BSD) still aren't doing SMP as well as other OS Care to elaborate? - Gilboa ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, 08 May 2011 17:28:07 +0300 Gilboa Davara gilb...@gmail.com wrote: On Sun, 2011-05-08 at 07:31 +, is...@zahav.net.il wrote: at this point Linux (and BSD) still aren't doing SMP as well as other OS Care to elaborate? I think it's well-known Solaris exploits multicore better than Linux or BSD. I see quite a few problem reports on the various BSD SMP mailing lists. I don't track Linux very much but I can see from conky on my boxes Linux just doesn't do that well. And race conditions are unfortunately an ongoing problem in many apps. I work on a different platform where multithreading and multiprocessing were a very early part of the design and I have seen a big difference in performance and lack of race conditions in that environment because it was based on a native multithreading model whereas UNIX was based on process forking and threading came much later and you could argue was not exactly implemented seamlessly. It's not an apples and apples comparison but the difference in software issues on those systems is night and day. As far as I can see those problems still haven't been resolved at the design or implementation levels. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, 2011-05-08 at 14:56 +, is...@zahav.net.il wrote: On Sun, 08 May 2011 17:28:07 +0300 Gilboa Davara gilb...@gmail.com wrote: On Sun, 2011-05-08 at 07:31 +, is...@zahav.net.il wrote: at this point Linux (and BSD) still aren't doing SMP as well as other OS Care to elaborate? I think it's well-known Solaris exploits multicore better than Linux or BSD. I have very little experience in BSD OS', so I can't really comment on that. However, in my own experience, the SMP performance of Linux vs. Solaris -greatly- depends on workload, with each having its own advantages and disadvantages. I don't track Linux very much but I can see from conky on my boxes Linux just doesn't do that well. And race conditions are unfortunately an ongoing problem in many apps. I don't think race condition means what you think it means... You're most likely mixing race condition and resource / lock contention. More-ever, you're mixing SMP kernel issues (Such as the soon-to-be-officially-dead BKL [big kernel lock] and SMP scheduler issues) and application design issues. (Read: Application that are not design with big/huge-SMP in mind) I work on a different platform where multithreading and multiprocessing were a very early part of the design and I have seen a big difference in performance and lack of race conditions in that environment because it was based on a native multithreading model whereas UNIX was based on process forking and threading came much later and you could argue was not exactly implemented seamlessly. It's not an apples and apples comparison but the difference in software issues on those systems is night and day. As far as I can see those problems still haven't been resolved at the design or implementation levels. A specific example will go a long way to explain your POV. - Gilboa ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, 08 May 2011 18:11:24 +0300 Gilboa Davara gilb...@gmail.com wrote: On Sun, 2011-05-08 at 14:56 +, is...@zahav.net.il wrote: On Sun, 08 May 2011 17:28:07 +0300 Gilboa Davara gilb...@gmail.com wrote: I don't track Linux very much but I can see from conky on my boxes Linux just doesn't do that well. And race conditions are unfortunately an ongoing problem in many apps. I don't think race condition means what you think it means... You're most likely mixing race condition and resource / lock contention. I'm not talking about contention which I understand they're trying to solve with the removal of the BKL, I'm talking about bugs in application code. But both contribute to software quality/usability issues for the end-user, especially with multicore processors. More-ever, you're mixing SMP kernel issues (Such as the soon-to-be-officially-dead BKL [big kernel lock] and SMP scheduler issues) and application design issues. (Read: Application that are not design with big/huge-SMP in mind) I'm not mixing anything, I'm saying *all* those things contribute to performance problems. I work on a different platform where multithreading and multiprocessing were a very early part of the design and I have seen a big difference in performance and lack of race conditions in that environment because it was based on a native multithreading model whereas UNIX was based on process forking and threading came much later and you could argue was not exactly implemented seamlessly. It's not an apples and apples comparison but the difference in software issues on those systems is night and day. As far as I can see those problems still haven't been resolved at the design or implementation levels. A specific example will go a long way to explain your POV. As I said my development experience is on a different platform with a fundamentally different design. In that system, process forking is very expensive and threading is very cheap- the opposite of the *NIX model. And there are three or so decades of symmetric SMP exploitation so that stuff is done and works and is not part of ongoing development and doesn't break or cause problems and most of the software is designed to protect against application level misuse and resource contention and deadlocks by killing offending work to keep the system running well. Those kinds of protections are not available in Linux or BSD AFAIK. For example you cannot spin the processor on System Z from an application for very long without being killed by the supervisor, but you can easily write userland code to hog the machine in *NIX. As *NIX was and is being changed to exploit SMP (and look at all the code that has been added let's say in the last 5 years to do this) it's very apparent to exploit the hardware threading is more useful than process forking. But that way of thinking is newish in *NIX and not all the tools and facilities (resource serialization etc) that are available in other systems (System Z for example) are available to *NIX so there are growing pains. A lot of progress has been made, no doubt. But there is still a lot of room for improvement. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, 2011-05-08 at 15:28 +, is...@zahav.net.il wrote: On Sun, 08 May 2011 18:11:24 +0300 Gilboa Davara gilb...@gmail.com wrote: On Sun, 2011-05-08 at 14:56 +, is...@zahav.net.il wrote: On Sun, 08 May 2011 17:28:07 +0300 Gilboa Davara gilb...@gmail.com wrote: I don't track Linux very much but I can see from conky on my boxes Linux just doesn't do that well. And race conditions are unfortunately an ongoing problem in many apps. I don't think race condition means what you think it means... You're most likely mixing race condition and resource / lock contention. I'm not talking about contention which I understand they're trying to solve with the removal of the BKL, I'm talking about bugs in application code. But both contribute to software quality/usability issues for the end-user, especially with multicore processors. More-ever, you're mixing SMP kernel issues (Such as the soon-to-be-officially-dead BKL [big kernel lock] and SMP scheduler issues) and application design issues. (Read: Application that are not design with big/huge-SMP in mind) I'm not mixing anything, I'm saying *all* those things contribute to performance problems. I work on a different platform where multithreading and multiprocessing were a very early part of the design and I have seen a big difference in performance and lack of race conditions in that environment because it was based on a native multithreading model whereas UNIX was based on process forking and threading came much later and you could argue was not exactly implemented seamlessly. It's not an apples and apples comparison but the difference in software issues on those systems is night and day. As far as I can see those problems still haven't been resolved at the design or implementation levels. A specific example will go a long way to explain your POV. As I said my development experience is on a different platform with a fundamentally different design. In that system, process forking is very expensive and threading is very cheap- the opposite of the *NIX model. And there are three or so decades of symmetric SMP exploitation so that stuff is done and works and is not part of ongoing development and doesn't break or cause problems and most of the software is designed to protect against application level misuse and resource contention and deadlocks by killing offending work to keep the system running well. Those kinds of protections are not available in Linux or BSD AFAIK. For example you cannot spin the processor on System Z from an application for very long without being killed by the supervisor, but you can easily write userland code to hog the machine in *NIX. As *NIX was and is being changed to exploit SMP (and look at all the code that has been added let's say in the last 5 years to do this) it's very apparent to exploit the hardware threading is more useful than process forking. But that way of thinking is newish in *NIX and not all the tools and facilities (resource serialization etc) that are available in other systems (System Z for example) are available to *NIX so there are growing pains. A lot of progress has been made, no doubt. But there is still a lot of room for improvement. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il and how is all this related to solaris Vs. linux? solaris is *nix, at least was the last time i heard ;) care to tell us the name of this operting system you are working on, instead of sounding so mysterious? is it a commercial general-purpose operating system? if so - what is it's name? or is it a proprietary system of the company you work for that does not work as a general-purpose operating system? when you say system Z - do you refer to what IBM formerly called MVS? --guy ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck? [OT]
On Sun, 08 May 2011 19:19:25 +0300 guy keren c...@actcom.co.il wrote: and how is all this related to solaris Vs. linux? solaris is *nix, at least was the last time i heard ;) Yes, you are right, but for some reason Solaris has the reputation for handling multicore better than Linux and BSD. Maybe you guys know why, it's not my area. I do use it and it has plusses and minuses like any other OS. I don't have a monster box to run it on yet so I can't confirm what I have been reading for the past few years. care to tell us the name of this operting system you are working on, instead of sounding so mysterious? is it a commercial general-purpose operating system? if so - what is it's name? or is it a proprietary system of the company you work for that does not work as a general-purpose operating system? when you say system Z - do you refer to what IBM formerly called MVS? Sorry this is going off topic, but just to answer your question ;-) Yes, no mystery...it's z/OS on z/Architecture (hardware platform). It's a very tightly coupled OS/platform that has been evolving since the 1960s. Extremely nice software/hardware environment to work with. It's a general purpose computing platform that is designed specifically around high throughput and RAS and has many design features in the OS and subsystems to make sure applications can't bang each other on the head too badly and can't do anything to system code at all. For example in the OLTP systems you can set various timers in the tuning parameters so that any deadlock or resource contention will be resolved by killing the offending party or waiter after the prescribed interval expires. It's granular and not an all-in-one setting. You can also set timers for elapsed or CPU time consumption by various work classes and kill runaway jobs, change their priority, etc. Anybody who wants to talk about it email me anytime. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, 2011-05-08 at 15:28 +, is...@zahav.net.il wroteך As I said my development experience is on a different platform with a fundamentally different design. In that system, process forking is very expensive and threading is very cheap- the opposite of the *NIX model. And there are three or so decades of symmetric SMP exploitation so that stuff is done and works and is not part of ongoing development and doesn't break or cause problems and most of the software is designed to protect against application level misuse and resource contention and deadlocks by killing offending work to keep the system running well. Those kinds of protections are not available in Linux or BSD AFAIK. For example you cannot spin the processor on System Z from an application for very long without being killed by the supervisor, but you can easily write userland code to hog the machine in *NIX. As *NIX was and is being changed to exploit SMP (and look at all the code that has been added let's say in the last 5 years to do this) it's very apparent to exploit the hardware threading is more useful than process forking. But that way of thinking is newish in *NIX and not all the tools and facilities (resource serialization etc) that are available in other systems (System Z for example) are available to *NIX so there are growing pains. A lot of progress has been made, no doubt. But there is still a lot of room for improvement. I can't say that in my experience pthread_x under 2.6 Linux is heavier or requires more resources than say, pthread or libthread under Solaris (though my experience is somewhat dated). Never the less, as I'm not sure we even agree on what the term SMP design means, lets agree not to agree... - Gilb ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On May 8, 2011, at 7:19 PM, guy keren wrote: when you say system Z - do you refer to what IBM formerly called MVS? IBM's had a lot of time to perfect it, their first multiprocessor machine was delivered in 1969. Geoff. -- Geoffrey S. Mendelson, N3OWJ/4X1GM Occam's Razor does not apply to electronics. If something won't turn on, it's not likely to be the power switch. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, 2011-05-08 at 09:47 +0300, Nadav Har'El wrote: On Sat, May 07, 2011, Omer Zak wrote about Re: Disk I/O as a bottleneck?: I suspect that speeding up /usr won't help improve performance that much. The applications, which seem to be sluggish, deal with a lot of user data in /home. Furthermore, this user data varies a lot with time, hence it is not that good idea to store it in SSD. As usual, it depends on your workload applies. In my own personal experience (and naturally, it might differ considerably from your use case), when I see sluggish behavior on a desktop machine, what is actually happening is that foreground activity, such as playing or working with video files, or such as compilation of a large project, causes a lot of other pages to be swapped out; And then, when you switch to a different application, it needs to swap pages in - either program text (code) directly from the executables, or data pages from the swap partition. I explicitly checked whether my PC is swapping or not (vmstat 1) and it was not. The sluggish behavior was not due to any RAM shortage. The contents of /proc/sys/vm/swappiness are: 60 I did find (using iostat -x -k 1) that both /usr and /home are heavily used (and %util hits 100% for both of them). In my PC they are in separate partitions but on the same hard disk (which is managed by LVM with encryption, except for a smallish boot partition). [OFFTOPIC] It was great experience to see all this thread resulting from my single posting. I can understand how it comes that some people enjoy trolling mailing lists. /wicked_grin --- Omer -- We live in a world of stupid people who enjoy being stupid. My own blog is at http://www.zak.co.il/tddpirate/ My opinions, as expressed in this E-mail message, are mine alone. They do not represent the official policy of any organization with which I may be affiliated in any way. WARNING TO SPAMMERS: at http://www.zak.co.il/spamwarning.html ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
[This E-mail message is bottom-posting contrary to my usual custom.] On Sun, 2011-05-08 at 17:26 +0300, Gilboa Davara wrote: On Sat, 2011-05-07 at 15:29 +0300, Omer Zak wrote: I have a PC with powerful processor, lots of RAM and SATA hard disk. Nevertheless I noticed that sometimes applications (evolution E-mail software and Firefox[iceweasel] Web browser) have the sluggish feel of a busy system (command line response time remains crisp, however, because the processor is 4x2 core one [4 cores, each multithreads as 2]). I run the gnome-system-monitor all the time. I notice that even when those applications feel sluggish, only one or at most two CPUs have high utilization, and there is plenty of free RAM (no swap space is used at all). Disk I/O is not monitored by gnome-system-monitor. So I suspect that the system is slowed down by disk I/O. I would like to eliminate it as a possible cause for the applications' sluggish feel. I ran smartctl tests on the hard disk, and they gave it clean bill of health. Therefore I/O error recovery should not be the reason for performance degradation. I am asking Collective Wisdom for advice about how to do: 1. Monitoring disk I/O load (counting I/O requests is not sufficient, as each request takes different time to complete due for example to disk head seeks or platter rotation time). 2. Disk scheduler fine-tuning possibilities to optimize disk I/O handling. 3. If smartctl is not sufficient to ensure that no I/O error overhead is incurred, how to better assess the hard disk's health? Thanks, --- Omer Hello Omer, Two questions: 1. Kernel version? 2.6.38 added a very small patch that that done wonders to eliminate foreground process scheduling issues that plagued desktop setups since the introduction of the CFS scheduler*. (Google for +2.6.38 +group +scheduling) On my Fedora 14 + 2.6.38 / dual 6C Xeon workstation I can easily watching a DVD movies while compiling the kernel and running a couple of VM's (using qemu-kvm) in the background. Doing the same using the stock Fedora 14 2.6.35 kernel is... err... interesting experience :) Standard Debian Squeeze kernel: $ uname -a Linux c4 2.6.32-5-vserver-amd64 #1 SMP Mon Mar 7 23:14:47 UTC 2011 x86_64 GNU/Linux 2. Are you sure your SATA is in AHCI mode? (Simply type: $ find /sys/devices | grep ahci) My SATA indeed runs in non-AHCI mode. The question now is how to configure it to run in AHCI mode? According to the Wikipedia article http://en.wikipedia.org/wiki/Advanced_Host_Controller_Interface, I need to modify a BIOS setting and maybe create a new initrd image. Another source: http://www.techmetica.com/howto/sata-ahci-mode-bios-setting.-what-does-it-do/ I can handle the BIOS setting but how to check whether a new initrd image is needed, and if necessary create it? --- Omer -- Bottom posters are filthy heretics and infidels and ought to be burned on the stake after having been tarred, feathered and having rotten eggs thrown at their faces! My own blog is at http://www.zak.co.il/tddpirate/ My opinions, as expressed in this E-mail message, are mine alone. They do not represent the official policy of any organization with which I may be affiliated in any way. WARNING TO SPAMMERS: at http://www.zak.co.il/spamwarning.html ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Disk I/O as a bottleneck?
I have a PC with powerful processor, lots of RAM and SATA hard disk. Nevertheless I noticed that sometimes applications (evolution E-mail software and Firefox[iceweasel] Web browser) have the sluggish feel of a busy system (command line response time remains crisp, however, because the processor is 4x2 core one [4 cores, each multithreads as 2]). I run the gnome-system-monitor all the time. I notice that even when those applications feel sluggish, only one or at most two CPUs have high utilization, and there is plenty of free RAM (no swap space is used at all). Disk I/O is not monitored by gnome-system-monitor. So I suspect that the system is slowed down by disk I/O. I would like to eliminate it as a possible cause for the applications' sluggish feel. I ran smartctl tests on the hard disk, and they gave it clean bill of health. Therefore I/O error recovery should not be the reason for performance degradation. I am asking Collective Wisdom for advice about how to do: 1. Monitoring disk I/O load (counting I/O requests is not sufficient, as each request takes different time to complete due for example to disk head seeks or platter rotation time). 2. Disk scheduler fine-tuning possibilities to optimize disk I/O handling. 3. If smartctl is not sufficient to ensure that no I/O error overhead is incurred, how to better assess the hard disk's health? Thanks, --- Omer -- My Commodore 64 is suffering from slowness and insufficiency of memory; and its display device is grievously short of pixels. Can anyone help? My own blog is at http://www.zak.co.il/tddpirate/ My opinions, as expressed in this E-mail message, are mine alone. They do not represent the official policy of any organization with which I may be affiliated in any way. WARNING TO SPAMMERS: at http://www.zak.co.il/spamwarning.html ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
you are stepping into never-never land ;) iostat -x -k 1 is your friend - just make sure you open a very wide terminal in which to look at it. disks are notoriously slow, regardless of error cases. it is enough if an applications perform a lot of random I/O - to make them work very slow. i'd refer you to the slides of the linux I/O lecture, at: http://haifux.org/lectures/254/alice_and_bob_in_io_land/ read them through. there are also some links to pages that discuss disk I/O tweaking. as for the elevator - you could try using the deadline elevator and see if this gives you any remedy. if you eventually decide that it is indeed disk I/O that slows you down, and if you have a lot of money to spend - you could consider buying an enterprise-grade SSD (e.g. from fusion I/O or from OCZ - although for your use-case, some of the cheaper SSDs will do) and use it instead of the hard disks. they only cost thousands of dollars for a 600GB SSD ;) --guy On Sat, 2011-05-07 at 15:29 +0300, Omer Zak wrote: I have a PC with powerful processor, lots of RAM and SATA hard disk. Nevertheless I noticed that sometimes applications (evolution E-mail software and Firefox[iceweasel] Web browser) have the sluggish feel of a busy system (command line response time remains crisp, however, because the processor is 4x2 core one [4 cores, each multithreads as 2]). I run the gnome-system-monitor all the time. I notice that even when those applications feel sluggish, only one or at most two CPUs have high utilization, and there is plenty of free RAM (no swap space is used at all). Disk I/O is not monitored by gnome-system-monitor. So I suspect that the system is slowed down by disk I/O. I would like to eliminate it as a possible cause for the applications' sluggish feel. I ran smartctl tests on the hard disk, and they gave it clean bill of health. Therefore I/O error recovery should not be the reason for performance degradation. I am asking Collective Wisdom for advice about how to do: 1. Monitoring disk I/O load (counting I/O requests is not sufficient, as each request takes different time to complete due for example to disk head seeks or platter rotation time). 2. Disk scheduler fine-tuning possibilities to optimize disk I/O handling. 3. If smartctl is not sufficient to ensure that no I/O error overhead is incurred, how to better assess the hard disk's health? Thanks, --- Omer ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sat, May 7, 2011 at 4:06 PM, guy keren c...@actcom.co.il wrote: you are stepping into never-never land ;) iostat -x -k 1 is your friend - just make sure you open a very wide terminal in which to look at it. disks are notoriously slow, regardless of error cases. it is enough if an applications perform a lot of random I/O - to make them work very slow. i'd refer you to the slides of the linux I/O lecture, at: http://haifux.org/lectures/254/alice_and_bob_in_io_land/ read them through. there are also some links to pages that discuss disk I/O tweaking. as for the elevator - you could try using the deadline elevator and see if this gives you any remedy. if you eventually decide that it is indeed disk I/O that slows you down, and if you have a lot of money to spend - you could consider buying an enterprise-grade SSD (e.g. from fusion I/O or from OCZ - although for your use-case, some of the cheaper SSDs will do) and use it instead of the hard disks. they only cost thousands of dollars for a 600GB SSD ;) Would probably be cheaper to get a bunch of SATAs into a raid array - spindle count matters after all. My home machine is not too new, but it definitely took wing after I replaced one large SATA disk with 6 smaller ones in a raid5 (I'm not risky enough for raid0) --guy On Sat, 2011-05-07 at 15:29 +0300, Omer Zak wrote: I have a PC with powerful processor, lots of RAM and SATA hard disk. Nevertheless I noticed that sometimes applications (evolution E-mail software and Firefox[iceweasel] Web browser) have the sluggish feel of a busy system (command line response time remains crisp, however, because the processor is 4x2 core one [4 cores, each multithreads as 2]). I run the gnome-system-monitor all the time. I notice that even when those applications feel sluggish, only one or at most two CPUs have high utilization, and there is plenty of free RAM (no swap space is used at all). Disk I/O is not monitored by gnome-system-monitor. So I suspect that the system is slowed down by disk I/O. I would like to eliminate it as a possible cause for the applications' sluggish feel. I ran smartctl tests on the hard disk, and they gave it clean bill of health. Therefore I/O error recovery should not be the reason for performance degradation. I am asking Collective Wisdom for advice about how to do: 1. Monitoring disk I/O load (counting I/O requests is not sufficient, as each request takes different time to complete due for example to disk head seeks or platter rotation time). 2. Disk scheduler fine-tuning possibilities to optimize disk I/O handling. 3. If smartctl is not sufficient to ensure that no I/O error overhead is incurred, how to better assess the hard disk's health? Thanks, --- Omer ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sat, 2011-05-07 at 16:19 +0300, Dima (Dan) Yasny wrote: On Sat, May 7, 2011 at 4:06 PM, guy keren c...@actcom.co.il wrote: you are stepping into never-never land ;) iostat -x -k 1 is your friend - just make sure you open a very wide terminal in which to look at it. disks are notoriously slow, regardless of error cases. it is enough if an applications perform a lot of random I/O - to make them work very slow. i'd refer you to the slides of the linux I/O lecture, at: http://haifux.org/lectures/254/alice_and_bob_in_io_land/ read them through. there are also some links to pages that discuss disk I/O tweaking. as for the elevator - you could try using the deadline elevator and see if this gives you any remedy. if you eventually decide that it is indeed disk I/O that slows you down, and if you have a lot of money to spend - you could consider buying an enterprise-grade SSD (e.g. from fusion I/O or from OCZ - although for your use-case, some of the cheaper SSDs will do) and use it instead of the hard disks. they only cost thousands of dollars for a 600GB SSD ;) Would probably be cheaper to get a bunch of SATAs into a raid array - spindle count matters after all. My home machine is not too new, but it definitely took wing after I replaced one large SATA disk with 6 smaller ones in a raid5 (I'm not risky enough for raid0) you are, of-course, quite right. provided that a hardware RAID controller is being used. --guy --guy On Sat, 2011-05-07 at 15:29 +0300, Omer Zak wrote: I have a PC with powerful processor, lots of RAM and SATA hard disk. Nevertheless I noticed that sometimes applications (evolution E-mail software and Firefox[iceweasel] Web browser) have the sluggish feel of a busy system (command line response time remains crisp, however, because the processor is 4x2 core one [4 cores, each multithreads as 2]). I run the gnome-system-monitor all the time. I notice that even when those applications feel sluggish, only one or at most two CPUs have high utilization, and there is plenty of free RAM (no swap space is used at all). Disk I/O is not monitored by gnome-system-monitor. So I suspect that the system is slowed down by disk I/O. I would like to eliminate it as a possible cause for the applications' sluggish feel. I ran smartctl tests on the hard disk, and they gave it clean bill of health. Therefore I/O error recovery should not be the reason for performance degradation. I am asking Collective Wisdom for advice about how to do: 1. Monitoring disk I/O load (counting I/O requests is not sufficient, as each request takes different time to complete due for example to disk head seeks or platter rotation time). 2. Disk scheduler fine-tuning possibilities to optimize disk I/O handling. 3. If smartctl is not sufficient to ensure that no I/O error overhead is incurred, how to better assess the hard disk's health? Thanks, --- Omer ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sat, May 7, 2011 at 4:41 PM, guy keren c...@actcom.co.il wrote: On Sat, 2011-05-07 at 16:19 +0300, Dima (Dan) Yasny wrote: On Sat, May 7, 2011 at 4:06 PM, guy keren c...@actcom.co.il wrote: you are stepping into never-never land ;) iostat -x -k 1 is your friend - just make sure you open a very wide terminal in which to look at it. disks are notoriously slow, regardless of error cases. it is enough if an applications perform a lot of random I/O - to make them work very slow. i'd refer you to the slides of the linux I/O lecture, at: http://haifux.org/lectures/254/alice_and_bob_in_io_land/ read them through. there are also some links to pages that discuss disk I/O tweaking. as for the elevator - you could try using the deadline elevator and see if this gives you any remedy. if you eventually decide that it is indeed disk I/O that slows you down, and if you have a lot of money to spend - you could consider buying an enterprise-grade SSD (e.g. from fusion I/O or from OCZ - although for your use-case, some of the cheaper SSDs will do) and use it instead of the hard disks. they only cost thousands of dollars for a 600GB SSD ;) Would probably be cheaper to get a bunch of SATAs into a raid array - spindle count matters after all. My home machine is not too new, but it definitely took wing after I replaced one large SATA disk with 6 smaller ones in a raid5 (I'm not risky enough for raid0) you are, of-course, quite right. provided that a hardware RAID controller is being used. I've seen performance increases even using fakeraid (the feared intel matrix 8.x in my machine) and mdadm. Especially in a machine that has a UPS (to step in instead of a BBU) and, like mentioned above - lots of crunchpower --guy --guy On Sat, 2011-05-07 at 15:29 +0300, Omer Zak wrote: I have a PC with powerful processor, lots of RAM and SATA hard disk. Nevertheless I noticed that sometimes applications (evolution E-mail software and Firefox[iceweasel] Web browser) have the sluggish feel of a busy system (command line response time remains crisp, however, because the processor is 4x2 core one [4 cores, each multithreads as 2]). I run the gnome-system-monitor all the time. I notice that even when those applications feel sluggish, only one or at most two CPUs have high utilization, and there is plenty of free RAM (no swap space is used at all). Disk I/O is not monitored by gnome-system-monitor. So I suspect that the system is slowed down by disk I/O. I would like to eliminate it as a possible cause for the applications' sluggish feel. I ran smartctl tests on the hard disk, and they gave it clean bill of health. Therefore I/O error recovery should not be the reason for performance degradation. I am asking Collective Wisdom for advice about how to do: 1. Monitoring disk I/O load (counting I/O requests is not sufficient, as each request takes different time to complete due for example to disk head seeks or platter rotation time). 2. Disk scheduler fine-tuning possibilities to optimize disk I/O handling. 3. If smartctl is not sufficient to ensure that no I/O error overhead is incurred, how to better assess the hard disk's health? Thanks, --- Omer ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
I would suggest making the check I mention in my own blog, in particular if you're running an old kernel. There has been a bug in the way the kernel handles heavy disk loads. http://billauer.co.il/blog/2010/10/disk-io-scheduler-load-dd-freeze-stall-hang/ Omer Zak wrote: I have a PC with powerful processor, lots of RAM and SATA hard disk. Nevertheless I noticed that sometimes applications (evolution E-mail software and Firefox[iceweasel] Web browser) have the sluggish feel of a busy system (command line response time remains crisp, however, because the processor is 4x2 core one [4 cores, each multithreads as 2]). I run the gnome-system-monitor all the time. I notice that even when those applications feel sluggish, only one or at most two CPUs have high utilization, and there is plenty of free RAM (no swap space is used at all). Disk I/O is not monitored by gnome-system-monitor. So I suspect that the system is slowed down by disk I/O. I would like to eliminate it as a possible cause for the applications' sluggish feel. I ran smartctl tests on the hard disk, and they gave it clean bill of health. Therefore I/O error recovery should not be the reason for performance degradation. I am asking Collective Wisdom for advice about how to do: 1. Monitoring disk I/O load (counting I/O requests is not sufficient, as each request takes different time to complete due for example to disk head seeks or platter rotation time). 2. Disk scheduler fine-tuning possibilities to optimize disk I/O handling. 3. If smartctl is not sufficient to ensure that no I/O error overhead is incurred, how to better assess the hard disk's health? Thanks, --- Omer -- Web: http://www.billauer.co.il ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
Hi Omer, On Sat, May 7, 2011 at 3:29 PM, Omer Zak w...@zak.co.il wrote: I have a PC with powerful processor, lots of RAM and SATA hard disk. Nevertheless I noticed that sometimes applications (evolution E-mail software and Firefox[iceweasel] Web browser) have the sluggish feel of a busy system (command line response time remains crisp, however, because the processor is 4x2 core one [4 cores, each multithreads as 2]). I run the gnome-system-monitor all the time. I notice that even when those applications feel sluggish, only one or at most two CPUs have high utilization, and there is plenty of free RAM (no swap space is used at all). The phenomena you are describing could be (though other explanations are still possible) a result of a completely different issue: Having 4 cores with 2 virtual cores on each means 8 threads could, theoretically, run in parallel. However, there is a large number of factors preventing 4 cores CPU from being 4 times faster than a single core CPU. The most common are, as of time being, the applications themselves - many of them are not written in a manner which enables them to use more than 1 CPU. Usually, it is either due to doing all of the CPU intensive work in one thread (i.e. the thread which renders the webpages to the screen in iceweasel) or due to coarse grain locking (i.e. locking the entire mozilla application every time we do garbage collection on the java-script heap, etc.). There are also issues of resource contention (i.e. firefox has a big working set, and therefore its speed is more affected by the amount of time it takes to access the RAM than it is affected by the CPU power - the CPU tends to mostly wait for data to arrive from the RAM), but that is less likely the case according to your description. When large amount of RAM is available, the chances that the sluggishness is due to slow disk access are rather slow - Linux is very aggressive about keeping everything in the cache, and if your application was spending most of its time waiting for disk I/O, you wouldn't have seen large CPU usage percentages - it was going into a sleep state until the data arrives, and wouldn't affect the user CPU usage percentage. To verify that, run top while the application is sluggish, and check if it the CPU usage is mostly in the user code (pressing 1 while top is running will show the per-CPU statistics in the top half of the screen, the percentage next to us is the relative amount of time the CPU spent running user code since the last sample). Good luck in hunting down your slowness. --Shachar ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sat, May 7, 2011 at 4:06 PM, guy keren c...@actcom.co.il wrote: if you eventually decide that it is indeed disk I/O that slows you down, and if you have a lot of money to spend - you could consider buying an enterprise-grade SSD (e.g. from fusion I/O or from OCZ - although for your use-case, some of the cheaper SSDs will do) and use it instead of the hard disks. they only cost thousands of dollars for a 600GB SSD ;) Is there a reason you're recommending such an expensive drives? I thought some time ago to buy a regular 40-80Gb and install the OS+swap there, and have a regular drive around for the rest of the data. Is there a reason this won't work? ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sat, 2011-05-07 at 21:49 +0300, Elazar Leibovich wrote: On Sat, May 7, 2011 at 4:06 PM, guy keren c...@actcom.co.il wrote: if you eventually decide that it is indeed disk I/O that slows you down, and if you have a lot of money to spend - you could consider buying an enterprise-grade SSD (e.g. from fusion I/O or from OCZ - although for your use-case, some of the cheaper SSDs will do) and use it instead of the hard disks. they only cost thousands of dollars for a 600GB SSD ;) Is there a reason you're recommending such an expensive drives? I thought some time ago to buy a regular 40-80Gb and install the OS +swap there, and have a regular drive around for the rest of the data. Is there a reason this won't work? I suspect that speeding up /usr won't help improve performance that much. The applications, which seem to be sluggish, deal with a lot of user data in /home. Furthermore, this user data varies a lot with time, hence it is not that good idea to store it in SSD. I liked more the idea of using a RAID scheme. --- Omer -- My Commodore 64 is suffering from slowness and insufficiency of memory; and its display device is grievously short of pixels. Can anyone help? My own blog is at http://www.zak.co.il/tddpirate/ My opinions, as expressed in this E-mail message, are mine alone. They do not represent the official policy of any organization with which I may be affiliated in any way. WARNING TO SPAMMERS: at http://www.zak.co.il/spamwarning.html ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sat, 2011-05-07 at 21:49 +0300, Elazar Leibovich wrote: On Sat, May 7, 2011 at 4:06 PM, guy keren c...@actcom.co.il wrote: if you eventually decide that it is indeed disk I/O that slows you down, and if you have a lot of money to spend - you could consider buying an enterprise-grade SSD (e.g. from fusion I/O or from OCZ - although for your use-case, some of the cheaper SSDs will do) and use it instead of the hard disks. they only cost thousands of dollars for a 600GB SSD ;) Is there a reason you're recommending such an expensive drives? I thought some time ago to buy a regular 40-80Gb and install the OS +swap there, and have a regular drive around for the rest of the data. Is there a reason this won't work? are you talking about using a low-end SSD? the problem with them, is that often their throughput for sequential operations is lower then that of normal hard disks. or are you talking about something different? --guy ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, May 8, 2011 at 12:20 AM, guy keren c...@actcom.co.il wrote: are you talking about using a low-end SSD? I'm actually not a big SSD expert, but I'm talking about relatively cheap SSD you can find in Ivory/Ksp, for instance Intel's http://www.zap.co.il/model.aspx?modelid=751136 the problem with them, is that often their throughput for sequential operations is lower then that of normal hard disks. Yeah, but what matters for the average user's computer speed is the random access speed, even if copying the 1Gb file will be a bit slower, when using the computer it'll be much faster, wouldn't it? ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, 2011-05-08 at 00:21 +0300, Elazar Leibovich wrote: On Sun, May 8, 2011 at 12:20 AM, guy keren c...@actcom.co.il wrote: are you talking about using a low-end SSD? I'm actually not a big SSD expert, but I'm talking about relatively cheap SSD you can find in Ivory/Ksp, for instance Intel's http://www.zap.co.il/model.aspx?modelid=751136 the problem with them, is that often their throughput for sequential operations is lower then that of normal hard disks. Yeah, but what matters for the average user's computer speed is the random access speed, even if copying the 1Gb file will be a bit slower, when using the computer it'll be much faster, wouldn't it? i guess the answer will be it depends :0 the fact is that a desktop user still does a lot of sequential I/O - so the sequential I/O speed still matters. another thing to note - the SSDs tend to start performing much worse if you fill them up to their max capacity. better use them in a lower capacity (e.g. 70-80% fill-factor), to keep their performance sane. i suggest that, once you get this drive, you come and tell us if you feel an improvement. then, once year after that - tell us again. --guy ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sat, May 07, 2011, guy keren wrote about Re: Disk I/O as a bottleneck?: and if you have a lot of money to spend - you could consider buying an enterprise-grade SSD (e.g. from fusion I/O or from OCZ - although for your use-case, some of the cheaper SSDs will do) and use it instead of the hard disks. they only cost thousands of dollars for a 600GB SSD ;) Instead of buying a huge SSD for thousands of dollars another option you might consider is to buy a relatively small SSD with just enough space to hold your / partition and swap space. Even 20 G may be enough. The rest of your disk - holding your source code, photos, songs, movies, or whatever you typically fill a terabyte with, will be a normal, cheap, hard disk. Several of my friends have gone with such a setup on their latest computer, and they are very pleased. -- Nadav Har'El|Sunday, May 8 2011, 4 Iyyar 5771 n...@math.technion.ac.il |- Phone +972-523-790466, ICQ 13349191 |My password is my dog's name. His name is http://nadav.harel.org.il |a#j!4@h, but I change it every month. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, 08 May 2011 07:28:49 +0300 Nadav Har'El n...@math.technion.ac.il wrote: Instead of buying a huge SSD for thousands of dollars another option you might consider is to buy a relatively small SSD with just enough space to hold your / partition and swap space. Even 20 G may be enough. The rest of your disk - holding your source code, photos, songs, movies, or whatever you typically fill a terabyte with, will be a normal, cheap, hard disk. I don't agree with this setup. Regular consumer drives setup with RAID to stripe are going to be much, much faster and have less problems in the long run than single SSDs at this point as well as being a better value until prices change a lot. Consider not using swap, because swap when in use causes a lot of thrashing and kills performance especially if you only have 1 or 2 drives. If you have a reasonably fast CPU (as the OP wrote) and more than 1G of RAM you can live without swap. Try it and like it: /sbin/swapoff -a Run for a few days and see if your performance doesn't improve. The only problem I can think of is if you run leaky code and don't have swap your system will lock up sooner. If you do have swap perhaps you will be able to see it coming. Another thing to consider is what filesystem(s) you use and what your mountpoints are. That's a religious debate and I'm not going to get into it except to say different filesystems have different advantages and disadvantages and it's worthwhile to do a bit of research to see which one(s) will work for you. You might consider one filesystem for volatile directories like /tmp and /var/log and others for relatively static filesystems like /. You should also consider your whole filesystem structure and the way you have mountpoints set up. If you have one big filesystem for everything (common but incorrect desktop config) you are not going to get the best performance possible. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On May 8, 2011, at 7:54 AM, is...@zahav.net.il wrote: I don't agree with this setup. Regular consumer drives setup with RAID to stripe are going to be much, much faster and have less problems in the long run than single SSDs at this point as well as being a better value until prices change a lot. If it's stuff you don't use often, or use sequentially, such as videos, cd/dvd software images, etc, you may consider USB drives. Be aware that using the NFS kernel server and USB disk drives causes kernel panics, lost data, etc. You can avoid the problem using samba shares or the user space NFS server. The user space NFS server is not compatible with some things, like RSYNC (missing function support), JDownloader (I/O on download directory) and the latest version of Ubuntu 11.04's gnome GUI (I/O error on home directory full eye candy turned on). With it turned off, it works. Between Kravitz, Bug and Office Depot, they occasionaly have disk wars where they sell USB external disks very cheaply. Geoff. -- Geoffrey S. Mendelson, N3OWJ/4X1GM Those who cannot remember the past are condemned to misquote it. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
