Re: Disk I/O as a bottleneck?
On Sun, May 08, 2011 at 07:28:49AM +0300, Nadav Har'El wrote: On Sat, May 07, 2011, guy keren wrote about Re: Disk I/O as a bottleneck?: and if you have a lot of money to spend - you could consider buying an enterprise-grade SSD (e.g. from fusion I/O or from OCZ - although for your use-case, some of the cheaper SSDs will do) and use it instead of the hard disks. they only cost thousands of dollars for a 600GB SSD ;) Instead of buying a huge SSD for thousands of dollars another option you might consider is to buy a relatively small SSD with just enough space to hold your / partition and swap space. Even 20 G may be enough. The rest of your disk - holding your source code, photos, songs, movies, or whatever you typically fill a terabyte with, will be a normal, cheap, hard disk. Several of my friends have gone with such a setup on their latest computer, and they are very pleased. I am considering, for my next laptop, and taking into account the fact that most laptops do not have space for two disks but do have some kind of flash memory slot (card reader) - usually SD-something, to have the OS on a (e.g.) SD card of 16 or 32 GB. I have no other experience with such cards, so I do not know if they are considered durable enough, fast enough - both random and sequential IO, both compared to SATA mechanical disks and to SATA flash ones, etc. Comments are welcome :-) -- Didi ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sat, May 07, 2011, Omer Zak wrote about Re: Disk I/O as a bottleneck?: I suspect that speeding up /usr won't help improve performance that much. The applications, which seem to be sluggish, deal with a lot of user data in /home. Furthermore, this user data varies a lot with time, hence it is not that good idea to store it in SSD. As usual, it depends on your workload applies. In my own personal experience (and naturally, it might differ considerably from your use case), when I see sluggish behavior on a desktop machine, what is actually happening is that foreground activity, such as playing or working with video files, or such as compilation of a large project, causes a lot of other pages to be swapped out; And then, when you switch to a different application, it needs to swap pages in - either program text (code) directly from the executables, or data pages from the swap partition. So when you switch to a GUI application, suddenly it takes a second to respond to a mouse click (it needs to swap in the relevant code and data), and when you type ls in a shell it takes much longer than usual (both the ls code and the directory are not in memory). Not only does fetching all these missing pages require a lot of seeks, which are slow on hard disks, it's even worse when that other application (which is using all that user data), continues to do a lot of seeks, and competes with the seeks needed to fetch the missing pages. In such a case if your system files - binaries, shared libraries, and swap, would be on a separate disk, everything might feel more responsive. If that separate disk had low seek times and hight throughput, it would be especially quick to recover from swap-outs, so you might see even better interactive behavior. Like I said, several of my friends tried this setup (SSD+hard disk) and liked the improved feel of the system (and the faster boot :-)). I haven't tried it myself, though. -- Nadav Har'El|Sunday, May 8 2011, 4 Iyyar 5771 n...@math.technion.ac.il |- Phone +972-523-790466, ICQ 13349191 |Christopher Robin Hood steals from the http://nadav.harel.org.il |rich and gives to the Pooh. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, May 08, 2011, is...@zahav.net.il wrote about Re: Disk I/O as a bottleneck?: I don't agree with this setup. Regular consumer drives setup with RAID to stripe are going to be much, much faster and have less problems in the long run than single SSDs at this point as well as being a better value until prices change a lot. Having two hard disks will, at best case, *double* your seek time. This is still pretty slow, isn't it? Won't an SSD, even cheap one, have a better random access read performance? Consider not using swap, because swap when in use causes a lot of thrashing and kills performance especially if you only have 1 or 2 drives. If you have Even without any swap space, you can have a lot of thrashing: Clean pages - program text (the code), shared libraries, memory-mapped files, and so on - are simply forgotten when the page cache is needed for other things, and when you get back to that other program, suddenly all its code and libraries are not in memory, and getting them back requires a lot of random-access reads from disk, and (especially if the disk is doing other things at the same time) causes the program to appear stuck or sluggish. Again, I don't know if this is the sort of problem that actually bothered the OP. -- Nadav Har'El|Sunday, May 8 2011, 4 Iyyar 5771 n...@math.technion.ac.il |- Phone +972-523-790466, ICQ 13349191 |The only intuitive interface is the http://nadav.harel.org.il |nipple. After that, it's all learned. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, May 8, 2011 at 7:28 AM, Nadav Har'El n...@math.technion.ac.ilwrote: Instead of buying a huge SSD for thousands of dollars another option you might consider is to buy a relatively small SSD with just enough space to hold your / partition and swap space. Even 20 G may be enough. The rest of your disk - holding your source code, photos, songs, movies, or whatever you typically fill a terabyte with, will be a normal, cheap, hard disk. Several of my friends have gone with such a setup on their latest computer, and they are very pleased. I have set up my latest system just like that. Though mine was a bit pricey: I went for the Intel X25-E 32GB. The OS and homedir are on it; Large datasets go on various Samsung SpinPoint 1TB F3 drives I've installed as well. The system is already more than a year old, and the free space is 20%, which, I am assuming, means I've already filled the disk (due to deletes and the SSD wear-leveling algorithms) and already doing erases, andthe performance is still nothing short of AMAZING - sub-1ms seek time is a great thing when you scan the filesystem etc. It just feels as if Disk I/O is no longer my bottleneck (and the CPU is a Quad Core AMD PhenomII 955 with 8GB RAM...). Of course - I don't use swap. Performance after 1 year as mentioned: # hdparm -t /dev/sda /dev/sda: Timing buffered disk reads: 722 MB in 3.00 seconds = 240.27 MB/sec As always, YMMV :) -- Shimi ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On May 8, 2011, at 9:30 AM, Yedidyah Bar-David wrote: I am considering, for my next laptop, and taking into account the fact that most laptops do not have space for two disks but do have some kind of flash memory slot (card reader) - usually SD-something, to have the OS on a (e.g.) SD card of 16 or 32 GB. I have no other experience with such cards, so I do not know if they are considered durable enough, fast enough - both random and sequential IO, both compared to SATA mechanical disks and to SATA flash ones, etc. Comments are welcome :-) It depends upon how you do it. The main difference in this case between a SOLID STATE DISK and memory card is the number of times you can write on it before it stops working. Modern memory cards do not use the same physical location for data all the time. The card itself randomizes where you write data, so that the useage of each bit on the card is spread out evenly. Of course this only works if the card is not full, and the emptier it is the better off you are. Whether this works with *NIX file systems is another question and I can't answer it. One of the bad things is that standard *NIX files systems are designed with magnetic media in mind, they update the access time of files every time you open them. This is bad for files that are opened often. The way around this is to mount a file system read only. Using a compressed read only file system, such as that on a live CD works well in this case. The problem with it is that you can't add software or change settings. UBUNTU has a setup where you can install a live system to a memory card/stick and it will mount your home directory in the unused space. If you can live with the limitations, then it will work for you. I think someone else said to use a small SSD for the system and a hard disk for your data. This would work extremely well for this situation where instead of a hard disk, you used a memory stick or card for it. It also depends upon what you are doing with it. Besides entertainment, my needs are fullfilled with an Xterm type terminal, SSH, a web browser and an email program. For entertainment, an MP3 player and one that will play 360P videos is enough. This can be accomplised with a lower power processor (Intel Atom for example) and a small screen. While you can get laptops with 15 inch screens and I7 processors, I'm not sure you would gain anything except a higher price by replacing a disk with an SSD/memory card combo. The latest Apple rumor is that they are going to produce a laptop soon with an ARM processor.Based on the success of the iPad, it probably will be a netbook size screen, a multicore ARM processor and a keyboard. It may or may not have a touch screen. I'm hoping that this rumor, whether there is any truth to it or not will fuel development of small ARM based netbooks. Unfortunately netbooks instead of getting smaller and cheaper, have gone the other way and become more expensive, larger, heavier and more powerfull. Geoff. -- Geoffrey S. Mendelson, N3OWJ/4X1GM Those who cannot remember the past are condemned to misquote it. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, May 8, 2011 at 10:02 AM, geoffrey mendelson geoffreymendel...@gmail.com wrote: One of the bad things is that standard *NIX files systems are designed with magnetic media in mind, they update the access time of files every time you open them. This is bad for files that are opened often. man mount search for 'noatime' -- Shimi ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, May 08, 2011, geoffrey mendelson wrote about Re: Disk I/O as a bottleneck?: One of the bad things is that standard *NIX files systems are designed with magnetic media in mind, they update the access time of files every time you open them. This is bad for files that are opened often. The way around this is to mount a file system read only. Using a Instead of mounting with the ro (read-only) option, a better option to use would be noatime, which disables just the updating of access-time. See also the nodiratime, relatime. These options are explained in mount(8). -- Nadav Har'El|Sunday, May 8 2011, 4 Iyyar 5771 n...@math.technion.ac.il |- Phone +972-523-790466, ICQ 13349191 |Happiness isn't getting what you want, http://nadav.harel.org.il |it's wanting what you've got. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, May 08, 2011, Nadav Har'El wrote about Re: Disk I/O as a bottleneck?: Having two hard disks will, at best case, *double* your seek time. This is Of course, I meant *half*, not *double* :-) -- Nadav Har'El|Sunday, May 8 2011, 4 Iyyar 5771 n...@math.technion.ac.il |- Phone +972-523-790466, ICQ 13349191 |I have an open mind - it's just closed http://nadav.harel.org.il |for repairs. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, 08 May 2011 09:55:35 +0300 Nadav Har'El n...@math.technion.ac.il wrote: On Sun, May 08, 2011, is...@zahav.net.il wrote about Re: Disk I/O as a bottleneck?: I don't agree with this setup. Regular consumer drives setup with RAID to stripe are going to be much, much faster and have less problems in the long run than single SSDs at this point as well as being a better value until prices change a lot. Having two hard disks will, at best case, *double* your seek time. This is still pretty slow, isn't it? Won't an SSD, even cheap one, have a better random access read performance? Striping works because the controllers can overlap I/O. There's no reason you have to seek on two drives, that's the *worst* case and probably shouldn't happen. The best case is the controller knows the data is on one drive and seek remains the same. Of course a physical drive is always more beneficial for sequential access than random, but striping is still better for random than not striping because of the caching- your system may be doing other work that needs data found on the other drive, and that means seek that you totally avoid. The more drives you spread your data across the less likely you are to thrash any one drive. As we say, six barbers, no waiting. I don't think any cheap SSDs cache enough to beat a good striped setup of even 2 physical disks and the page sizes aren't usually matched to Linux's requirements. I read an article about how SSD design was mostly made to Windows filesystems and can work against you somewhat if you use them on Linux or other OS. I don't know how true it was but it makes you think. Anyway as you add more disks you keep increasing performance with a striping setup, there is no downside except if you mirror but then again there is no free lunch. You can have it fast or reliable or cheap, but you can't have it all. The question is how much RAID and how many drives you can buy for the price of a good SSD, how much cache will you get with striped RAID and how much cache do you get with a good SSD, and how long will everything last? SSD still has some nasty issues like wear leveling, lifetime problems, etc. Enterprise servers are still using physical drives. Those guys can pay anything and they buy the old stuff so that must tell us something. Personally I'm staying with drives that spin until technology improves and I start reading about production servers that are totally SSD. YMMV ;-) Consider not using swap, because swap when in use causes a lot of thrashing and kills performance especially if you only have 1 or 2 drives. If you have Even without any swap space, you can have a lot of thrashing: Clean pages - program text (the code), shared libraries, memory-mapped files, and so on - are simply forgotten when the page cache is needed for other things, and when you get back to that other program, suddenly all its code and libraries are not in memory, and getting them back requires a lot of random-access reads from disk, and (especially if the disk is doing other things at the same time) causes the program to appear stuck or sluggish. That's true but swap only makes it worse since it's many magnitudes slower than RAM and in my experience, Linux doesn't do a very good job of managing it. If I have 4G of RAM and less than 2G is in use, I still see a few meg being used by swap. What's the reason for that? I don't want to start swapping until my RAM is very close to maxxed out. And then I want swap to get cleaned out very quickly after RAM is available. Swapping should be an absolute last resort on a modern machine. It looks like swap processing is something that hasn't received much attention since it was designed and things have changed a lot and it's time to stop using it until they revisit how it should work. If you have a monitoring program that tells you how much RAM and swap are in use, you will see a dramatic slowdown in performance when you swap even a little. I think the OP said he had a fast CPU and tons of RAM so if the system feels sluggish then the obvious things to look at are filesystem layout, choice of filesystems, turning off swap. If that doesn't work then maybe look into another OS...at this point Linux (and BSD) still aren't doing SMP as well as other OS but there may not be any better choices on Intel hardware. As you or somebody else pointed out, many apps don't thread enough to exploit a hyperthreading quad. Maybe the OP should look into running Solaris, which is known to do well in this area. It's probably going to be a long time until Linux and Linux apps really exploit multicore well. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Upgrading Android Under Linux
On Fri, May 6, 2011 at 17:08, Shachar Shemesh shac...@shemesh.biz wrote: What is NB? http://en.wikipedia.org/wiki/Nota_bene Gratias tibi ago! -- Dotan Cohen http://gibberish.co.il http://what-is-what.com ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, 08 May 2011 10:02:18 +0300 geoffrey mendelson geoffreymendel...@gmail.com wrote: One of the bad things is that standard *NIX files systems are designed with magnetic media in mind, they update the access time of files every time you open them. This is bad for files that are opened often. You can stop this with noatime in the fstab. It's something I do whenever I set up a new install. The way around this is to mount a file system read only. Using a compressed read only file system, such as that on a live CD works well in this case. The problem with it is that you can't add software or change settings. You don't need to mount the fs r/o to use noatime. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, 08 May 2011 10:15:30 +0300 Nadav Har'El n...@math.technion.ac.il wrote: On Sun, May 08, 2011, Nadav Har'El wrote about Re: Disk I/O as a bottleneck?: Having two hard disks will, at best case, *double* your seek time. This is Of course, I meant *half*, not *double* :-) Actually it might do a lot better than that because of caching. Aside from the fact striped data is spread across drives, it exploits caching better since data from more than one MRU file is likely to be in cache. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, 2011-05-08 at 09:57 +0300, shimi wrote: On Sun, May 8, 2011 at 7:28 AM, Nadav Har'El n...@math.technion.ac.il wrote: Instead of buying a huge SSD for thousands of dollars another option you might consider is to buy a relatively small SSD with just enough space to hold your / partition and swap space. Even 20 G may be enough. The rest of your disk - holding your source code, photos, songs, movies, or whatever you typically fill a terabyte with, will be a normal, cheap, hard disk. Several of my friends have gone with such a setup on their latest computer, and they are very pleased. I have set up my latest system just like that. Though mine was a bit pricey: I went for the Intel X25-E 32GB. The OS and homedir are on it; Large datasets go on various Samsung SpinPoint 1TB F3 drives I've installed as well. The system is already more than a year old, and the free space is 20%, which, I am assuming, means I've already filled the disk (due to deletes and the SSD wear-leveling algorithms) and already doing erases, andthe performance is still nothing short of AMAZING - sub-1ms seek time is a great thing when you scan the filesystem etc. It just feels as if Disk I/O is no longer my bottleneck (and the CPU is a Quad Core AMD PhenomII 955 with 8GB RAM...). Of course - I don't use swap. Performance after 1 year as mentioned: # hdparm -t /dev/sda /dev/sda: Timing buffered disk reads: 722 MB in 3.00 seconds = 240.27 MB/sec As always, YMMV :) what tends to get worse after the SSD becomes full is writes, not reads. and combinations of reads and writes make things look worse (the writes slow down the reads). however, if you feel that the system is very fast after one year of use - that's good enough for me. do you have the ability to extract wear leveling information from your SSD? it would be interesting to know whether the drive is being used in a manner that will indeed comply with the life-time expentency it is sold with (5 years?), or better, or worse. --guy ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, 2011-05-08 at 09:30 +0300, Yedidyah Bar-David wrote: On Sun, May 08, 2011 at 07:28:49AM +0300, Nadav Har'El wrote: On Sat, May 07, 2011, guy keren wrote about Re: Disk I/O as a bottleneck?: and if you have a lot of money to spend - you could consider buying an enterprise-grade SSD (e.g. from fusion I/O or from OCZ - although for your use-case, some of the cheaper SSDs will do) and use it instead of the hard disks. they only cost thousands of dollars for a 600GB SSD ;) Instead of buying a huge SSD for thousands of dollars another option you might consider is to buy a relatively small SSD with just enough space to hold your / partition and swap space. Even 20 G may be enough. The rest of your disk - holding your source code, photos, songs, movies, or whatever you typically fill a terabyte with, will be a normal, cheap, hard disk. Several of my friends have gone with such a setup on their latest computer, and they are very pleased. I am considering, for my next laptop, and taking into account the fact that most laptops do not have space for two disks but do have some kind of flash memory slot (card reader) - usually SD-something, to have the OS on a (e.g.) SD card of 16 or 32 GB. I have no other experience with such cards, so I do not know if they are considered durable enough, fast enough - both random and sequential IO, both compared to SATA mechanical disks and to SATA flash ones, etc. Comments are welcome :-) SD cards are much much much slower then SSDs with regards to sequential I/O - and i think they live a shorter life. if you want to use one - you should make sure it's set in a mostly read-only setup. problematic directories include /var and /tmp. specifically, installing RPMs on such drives works much slower then on hard disks. they are still used on various appliances and embedded systems, in such a mostly read-only configuration. --guy ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, May 8, 2011 at 12:01 PM, guy keren c...@actcom.co.il wrote: On Sun, 2011-05-08 at 09:57 +0300, shimi wrote: what tends to get worse after the SSD becomes full is writes, not reads. and combinations of reads and writes make things look worse (the writes slow down the reads). You're of course correct. Hope this satisfies the issue: $ cat test.sh #!/bin/sh dd if=/dev/zero of=test123456.dat bs=10 count=1 sync $ time ./test.sh 1+0 records in 1+0 records out 10 bytes (1.0 GB) copied, 3.89247 s, 257 MB/s real0m6.158s user0m0.001s sys 0m1.738s (obviously dd itself has stuff in RAM. this is why I used time with sync after the dd. 1GB in 6.158 seconds is 162MB/s not too bad. still better than the Samsung F3 which is one of the fastest disks out there... same script on that 1TB drive takes 12.239s to complete the same task..) however, if you feel that the system is very fast after one year of use - that's good enough for me. I do. And I don't think it's such a difference. Most writes are pretty small, and will not halt the system. I think most of the time the system is slow due to the heads busy with moving around the platter (seek), something that is almost completely eliminated in SSD - and *that's* why you have the performance boost. Correct, there are lousy SSDs that write very slowly, and then block I/O to the lengthy erase process, and will hang the app or the whole bus (depends on the controller, I guess?)... but I don't think the X25-E falls under that category :) do you have the ability to extract wear leveling information from your SSD? it would be interesting to know whether the drive is being used in a manner that will indeed comply with the life-time expentency it is sold with (5 years?), or better, or worse. I don't know, how do you extract such information? The rated MTBF of my specific drive is 2 million hours. If I still know my math, that's some 228 years -- Shimi ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On The rated MTBF of my specific drive is 2 million hours. If I still know my math, that's some 228 years Which is meaningless. The life expectency of a drive is closer to the length of the warranty period. Warranties are decided based upon projected return rates. The manufacturers want no more than a 5% return rate, some less such as 2% or 3%. Once they expect more to come back, they no longer provide a warranty. So if they expect that 2% will come back in the first 3 years. They give you a 3 year warranty. These warranties only apply to retail drives. OEM drives generally are sold without a warranty at all. The OEM provides a warranty for the entire system, and negotiates a lower price with the understanding that they will eat any returns in exchange. That's even starting to affect computers, I saw in Friday's Yediot a computer sold for 15% less if you took it with a one year warranty instead of a three year one. Since it was a low end computer, possibly obsolete in a year or two, it may have been worth it. On the other hand my son is chomping at the bit for the one year warranty to expire on his computer so he can talk all of his relatives into chipping in and buying him a new video card. Last year's high end video card from last year is not fast enough now. :-) I on the other hand was recently given a five year old computer, which due to memory restrictions and a lousy BIOS will be permanently stuck on Windows XP. A new hard drive, a fresh install of Linux, and I'm happy. I/O is absymal, but the CPU is fast. Geoff. -- Geoffrey S. Mendelson, N3OWJ/4X1GM Those who cannot remember the past are condemned to misquote it. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, 2011-05-08 at 12:26 +0300, shimi wrote: On Sun, May 8, 2011 at 12:01 PM, guy keren c...@actcom.co.il wrote: On Sun, 2011-05-08 at 09:57 +0300, shimi wrote: what tends to get worse after the SSD becomes full is writes, not reads. and combinations of reads and writes make things look worse (the writes slow down the reads). You're of course correct. Hope this satisfies the issue: $ cat test.sh #!/bin/sh dd if=/dev/zero of=test123456.dat bs=10 count=1 sync $ time ./test.sh 1+0 records in 1+0 records out 10 bytes (1.0 GB) copied, 3.89247 s, 257 MB/s real0m6.158s user0m0.001s sys 0m1.738s (obviously dd itself has stuff in RAM. this is why I used time with sync after the dd. 1GB in 6.158 seconds is 162MB/s not too bad. still better than the Samsung F3 which is one of the fastest disks out there... same script on that 1TB drive takes 12.239s to complete the same task..) however, if you feel that the system is very fast after one year of use - that's good enough for me. I do. And I don't think it's such a difference. Most writes are pretty small, and will not halt the system. I think most of the time the system is slow due to the heads busy with moving around the platter (seek), something that is almost completely eliminated in SSD - and *that's* why you have the performance boost. Correct, there are lousy SSDs that write very slowly, and then block I/O to the lengthy erase process, and will hang the app or the whole bus (depends on the controller, I guess?)... but I don't think the X25-E falls under that category :) do you have the ability to extract wear leveling information from your SSD? it would be interesting to know whether the drive is being used in a manner that will indeed comply with the life-time expentency it is sold with (5 years?), or better, or worse. I don't know, how do you extract such information? The rated MTBF of my specific drive is 2 million hours. If I still know my math, that's some 228 years -- Shimi wear leveling has nothing to do with MTBF. once you write ~100,000 times to a single cell in the SSD - it's dead. due to the wear leveling methods of the SSD - this will happen once you write ~100,000 times to all cell groups on the SSD - assuming the wear-leveling algorithm of the SSD is implemented without glitches. note that these writes don't come only from the host - many of them are generated internally by the SSD, due to its wear-leveling algorithms. an SSD could perform several writes for each host-initiated write operation on average. intel claims their X25-E has very impressive algorithms in this regard. it'll be interesting to check these claims with the actual state of your SSD. fetching wear-leveling info is SSD-dependent. you'll need to check if intel provides a tool to do that on linux, for your SSD. --guy ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, May 8, 2011 at 1:21 PM, guy keren c...@actcom.co.il wrote: On Sun, 2011-05-08 at 12:26 +0300, shimi wrote: On Sun, May 8, 2011 at 12:01 PM, guy keren c...@actcom.co.il wrote: do you have the ability to extract wear leveling information from your SSD? it would be interesting to know whether the drive is being used in a manner that will indeed comply with the life-time expentency it is sold with (5 years?), or better, or worse. I don't know, how do you extract such information? The rated MTBF of my specific drive is 2 million hours. If I still know my math, that's some 228 years -- Shimi wear leveling has nothing to do with MTBF. once you write ~100,000 times to a single cell in the SSD - it's dead. due to the wear leveling methods of the SSD - this will happen once you write ~100,000 times to all cell groups on the SSD - assuming the wear-leveling algorithm of the SSD is implemented without glitches. I know... I was referring more to the life time expectancy it is sold with when I quoted that number. Unless block write fails do not consist of a failure, and if not, I don't know what is, in an SSD :). Obviously on a DB data disk it is going to happen much faster than on my desktop. b.t.w. IIRC when a cell dies, it does so gracefully; I.e. no data is lost, and there are spare blocks for that case... and even when they're all full, you just get to the point that you still have your data read-only. I vaguely remember I read that somewhere... and if it's indeed like that, this is still way better than a regular hard drive - those tend to usually take all your data with them, and are much more sensitive to many things (shock - physical/electric, heat, etc...) note that these writes don't come only from the host - many of them are generated internally by the SSD, due to its wear-leveling algorithms. an SSD could perform several writes for each host-initiated write operation on average. intel claims their X25-E has very impressive algorithms in this regard. it'll be interesting to check these claims with the actual state of your SSD. I know that too :) fetching wear-leveling info is SSD-dependent. you'll need to check if intel provides a tool to do that on linux, for your SSD. Google didn't help me a lot in that regard (I understand there's a Win$ SW maybe to do that; Unfortunately, I don't have code from Redmond anywhere inside my chassis.) So if you find something yourself, I'm willing to test... -- Shimi ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
SSD? (Re: Disk I/O as a bottleneck?)
On Sun, 2011-05-08 at 13:27 +0300, shimi wrote: b.t.w. IIRC when a cell dies, it does so gracefully; I.e. no data is lost, and there are spare blocks for that case... and even when they're all full, you just get to the point that you still have your data read-only. I vaguely remember I read that somewhere... and if it's indeed like that, this is still way better than a regular hard drive - those tend to usually take all your data with them, and are much more sensitive to many things (shock - physical/electric, heat, etc...) While its very unlikely that you'll ever come close to reaching the (more-or-less-theoretical) per-cell write limit, SSD's do have their share of major issues that should be taken into account: 1. Bricking: As a general rule, SSD's are -far- more vulnerable to data corruption and data loss issues and the reason for this is rather simple: Compared to spinning HDD's, SSD's have highly complex firmware that uses complex write-leveling algorithms and, in some cases, complex compression schemes. (SandForce SF-1xxx, SF-2xxx controllers) As a result, any corner bug in the SSD's firmware will usually brick your SSD, leaving you with nothing. Far worse, while we (as an industry) have extensive of experience in yanking data out of a damaged spinning HDD's, there's far less collective experience in dealing with bricked SDD's. 2. Experience: We literally have decades of experience with dealing with spinning HDD's. Give me a RAID with 1,000 drives and I can predict how many drives will die each month, each year. On the other hand, we simply do not have sufficient experience in dealing with high-end SSD's to draw the predict the same when it comes to SSD's. 3. Performance: TRIM, file-system optimization, dealing with drives running at 90% full capacity, etc. Getting the most out of an SSD's requires careful planning. Let me be clear: I do not doubt that SSD's will slowly replace spinning drives in 90% of all the use cases. I am saying that I'd consider twice before using SDD's in mission critical environment on in places with inadequate backup strategy. - Gilboa ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sat, 2011-05-07 at 15:29 +0300, Omer Zak wrote: I have a PC with powerful processor, lots of RAM and SATA hard disk. Nevertheless I noticed that sometimes applications (evolution E-mail software and Firefox[iceweasel] Web browser) have the sluggish feel of a busy system (command line response time remains crisp, however, because the processor is 4x2 core one [4 cores, each multithreads as 2]). I run the gnome-system-monitor all the time. I notice that even when those applications feel sluggish, only one or at most two CPUs have high utilization, and there is plenty of free RAM (no swap space is used at all). Disk I/O is not monitored by gnome-system-monitor. So I suspect that the system is slowed down by disk I/O. I would like to eliminate it as a possible cause for the applications' sluggish feel. I ran smartctl tests on the hard disk, and they gave it clean bill of health. Therefore I/O error recovery should not be the reason for performance degradation. I am asking Collective Wisdom for advice about how to do: 1. Monitoring disk I/O load (counting I/O requests is not sufficient, as each request takes different time to complete due for example to disk head seeks or platter rotation time). 2. Disk scheduler fine-tuning possibilities to optimize disk I/O handling. 3. If smartctl is not sufficient to ensure that no I/O error overhead is incurred, how to better assess the hard disk's health? Thanks, --- Omer Hello Omer, Two questions: 1. Kernel version? 2.6.38 added a very small patch that that done wonders to eliminate foreground process scheduling issues that plagued desktop setups since the introduction of the CFS scheduler*. (Google for +2.6.38 +group +scheduling) On my Fedora 14 + 2.6.38 / dual 6C Xeon workstation I can easily watching a DVD movies while compiling the kernel and running a couple of VM's (using qemu-kvm) in the background. Doing the same using the stock Fedora 14 2.6.35 kernel is... err... interesting experience :) 2. Are you sure your SATA is in AHCI mode? (Simply type: $ find /sys/devices | grep ahci) - Gilboa * At least in my case... ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, 2011-05-08 at 07:31 +, is...@zahav.net.il wrote: at this point Linux (and BSD) still aren't doing SMP as well as other OS Care to elaborate? - Gilboa ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, 08 May 2011 17:28:07 +0300 Gilboa Davara gilb...@gmail.com wrote: On Sun, 2011-05-08 at 07:31 +, is...@zahav.net.il wrote: at this point Linux (and BSD) still aren't doing SMP as well as other OS Care to elaborate? I think it's well-known Solaris exploits multicore better than Linux or BSD. I see quite a few problem reports on the various BSD SMP mailing lists. I don't track Linux very much but I can see from conky on my boxes Linux just doesn't do that well. And race conditions are unfortunately an ongoing problem in many apps. I work on a different platform where multithreading and multiprocessing were a very early part of the design and I have seen a big difference in performance and lack of race conditions in that environment because it was based on a native multithreading model whereas UNIX was based on process forking and threading came much later and you could argue was not exactly implemented seamlessly. It's not an apples and apples comparison but the difference in software issues on those systems is night and day. As far as I can see those problems still haven't been resolved at the design or implementation levels. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, 2011-05-08 at 14:56 +, is...@zahav.net.il wrote: On Sun, 08 May 2011 17:28:07 +0300 Gilboa Davara gilb...@gmail.com wrote: On Sun, 2011-05-08 at 07:31 +, is...@zahav.net.il wrote: at this point Linux (and BSD) still aren't doing SMP as well as other OS Care to elaborate? I think it's well-known Solaris exploits multicore better than Linux or BSD. I have very little experience in BSD OS', so I can't really comment on that. However, in my own experience, the SMP performance of Linux vs. Solaris -greatly- depends on workload, with each having its own advantages and disadvantages. I don't track Linux very much but I can see from conky on my boxes Linux just doesn't do that well. And race conditions are unfortunately an ongoing problem in many apps. I don't think race condition means what you think it means... You're most likely mixing race condition and resource / lock contention. More-ever, you're mixing SMP kernel issues (Such as the soon-to-be-officially-dead BKL [big kernel lock] and SMP scheduler issues) and application design issues. (Read: Application that are not design with big/huge-SMP in mind) I work on a different platform where multithreading and multiprocessing were a very early part of the design and I have seen a big difference in performance and lack of race conditions in that environment because it was based on a native multithreading model whereas UNIX was based on process forking and threading came much later and you could argue was not exactly implemented seamlessly. It's not an apples and apples comparison but the difference in software issues on those systems is night and day. As far as I can see those problems still haven't been resolved at the design or implementation levels. A specific example will go a long way to explain your POV. - Gilboa ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, 08 May 2011 18:11:24 +0300 Gilboa Davara gilb...@gmail.com wrote: On Sun, 2011-05-08 at 14:56 +, is...@zahav.net.il wrote: On Sun, 08 May 2011 17:28:07 +0300 Gilboa Davara gilb...@gmail.com wrote: I don't track Linux very much but I can see from conky on my boxes Linux just doesn't do that well. And race conditions are unfortunately an ongoing problem in many apps. I don't think race condition means what you think it means... You're most likely mixing race condition and resource / lock contention. I'm not talking about contention which I understand they're trying to solve with the removal of the BKL, I'm talking about bugs in application code. But both contribute to software quality/usability issues for the end-user, especially with multicore processors. More-ever, you're mixing SMP kernel issues (Such as the soon-to-be-officially-dead BKL [big kernel lock] and SMP scheduler issues) and application design issues. (Read: Application that are not design with big/huge-SMP in mind) I'm not mixing anything, I'm saying *all* those things contribute to performance problems. I work on a different platform where multithreading and multiprocessing were a very early part of the design and I have seen a big difference in performance and lack of race conditions in that environment because it was based on a native multithreading model whereas UNIX was based on process forking and threading came much later and you could argue was not exactly implemented seamlessly. It's not an apples and apples comparison but the difference in software issues on those systems is night and day. As far as I can see those problems still haven't been resolved at the design or implementation levels. A specific example will go a long way to explain your POV. As I said my development experience is on a different platform with a fundamentally different design. In that system, process forking is very expensive and threading is very cheap- the opposite of the *NIX model. And there are three or so decades of symmetric SMP exploitation so that stuff is done and works and is not part of ongoing development and doesn't break or cause problems and most of the software is designed to protect against application level misuse and resource contention and deadlocks by killing offending work to keep the system running well. Those kinds of protections are not available in Linux or BSD AFAIK. For example you cannot spin the processor on System Z from an application for very long without being killed by the supervisor, but you can easily write userland code to hog the machine in *NIX. As *NIX was and is being changed to exploit SMP (and look at all the code that has been added let's say in the last 5 years to do this) it's very apparent to exploit the hardware threading is more useful than process forking. But that way of thinking is newish in *NIX and not all the tools and facilities (resource serialization etc) that are available in other systems (System Z for example) are available to *NIX so there are growing pains. A lot of progress has been made, no doubt. But there is still a lot of room for improvement. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, 2011-05-08 at 15:28 +, is...@zahav.net.il wrote: On Sun, 08 May 2011 18:11:24 +0300 Gilboa Davara gilb...@gmail.com wrote: On Sun, 2011-05-08 at 14:56 +, is...@zahav.net.il wrote: On Sun, 08 May 2011 17:28:07 +0300 Gilboa Davara gilb...@gmail.com wrote: I don't track Linux very much but I can see from conky on my boxes Linux just doesn't do that well. And race conditions are unfortunately an ongoing problem in many apps. I don't think race condition means what you think it means... You're most likely mixing race condition and resource / lock contention. I'm not talking about contention which I understand they're trying to solve with the removal of the BKL, I'm talking about bugs in application code. But both contribute to software quality/usability issues for the end-user, especially with multicore processors. More-ever, you're mixing SMP kernel issues (Such as the soon-to-be-officially-dead BKL [big kernel lock] and SMP scheduler issues) and application design issues. (Read: Application that are not design with big/huge-SMP in mind) I'm not mixing anything, I'm saying *all* those things contribute to performance problems. I work on a different platform where multithreading and multiprocessing were a very early part of the design and I have seen a big difference in performance and lack of race conditions in that environment because it was based on a native multithreading model whereas UNIX was based on process forking and threading came much later and you could argue was not exactly implemented seamlessly. It's not an apples and apples comparison but the difference in software issues on those systems is night and day. As far as I can see those problems still haven't been resolved at the design or implementation levels. A specific example will go a long way to explain your POV. As I said my development experience is on a different platform with a fundamentally different design. In that system, process forking is very expensive and threading is very cheap- the opposite of the *NIX model. And there are three or so decades of symmetric SMP exploitation so that stuff is done and works and is not part of ongoing development and doesn't break or cause problems and most of the software is designed to protect against application level misuse and resource contention and deadlocks by killing offending work to keep the system running well. Those kinds of protections are not available in Linux or BSD AFAIK. For example you cannot spin the processor on System Z from an application for very long without being killed by the supervisor, but you can easily write userland code to hog the machine in *NIX. As *NIX was and is being changed to exploit SMP (and look at all the code that has been added let's say in the last 5 years to do this) it's very apparent to exploit the hardware threading is more useful than process forking. But that way of thinking is newish in *NIX and not all the tools and facilities (resource serialization etc) that are available in other systems (System Z for example) are available to *NIX so there are growing pains. A lot of progress has been made, no doubt. But there is still a lot of room for improvement. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il and how is all this related to solaris Vs. linux? solaris is *nix, at least was the last time i heard ;) care to tell us the name of this operting system you are working on, instead of sounding so mysterious? is it a commercial general-purpose operating system? if so - what is it's name? or is it a proprietary system of the company you work for that does not work as a general-purpose operating system? when you say system Z - do you refer to what IBM formerly called MVS? --guy ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck? [OT]
On Sun, 08 May 2011 19:19:25 +0300 guy keren c...@actcom.co.il wrote: and how is all this related to solaris Vs. linux? solaris is *nix, at least was the last time i heard ;) Yes, you are right, but for some reason Solaris has the reputation for handling multicore better than Linux and BSD. Maybe you guys know why, it's not my area. I do use it and it has plusses and minuses like any other OS. I don't have a monster box to run it on yet so I can't confirm what I have been reading for the past few years. care to tell us the name of this operting system you are working on, instead of sounding so mysterious? is it a commercial general-purpose operating system? if so - what is it's name? or is it a proprietary system of the company you work for that does not work as a general-purpose operating system? when you say system Z - do you refer to what IBM formerly called MVS? Sorry this is going off topic, but just to answer your question ;-) Yes, no mystery...it's z/OS on z/Architecture (hardware platform). It's a very tightly coupled OS/platform that has been evolving since the 1960s. Extremely nice software/hardware environment to work with. It's a general purpose computing platform that is designed specifically around high throughput and RAS and has many design features in the OS and subsystems to make sure applications can't bang each other on the head too badly and can't do anything to system code at all. For example in the OLTP systems you can set various timers in the tuning parameters so that any deadlock or resource contention will be resolved by killing the offending party or waiter after the prescribed interval expires. It's granular and not an all-in-one setting. You can also set timers for elapsed or CPU time consumption by various work classes and kill runaway jobs, change their priority, etc. Anybody who wants to talk about it email me anytime. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, 2011-05-08 at 15:28 +, is...@zahav.net.il wroteŚ As I said my development experience is on a different platform with a fundamentally different design. In that system, process forking is very expensive and threading is very cheap- the opposite of the *NIX model. And there are three or so decades of symmetric SMP exploitation so that stuff is done and works and is not part of ongoing development and doesn't break or cause problems and most of the software is designed to protect against application level misuse and resource contention and deadlocks by killing offending work to keep the system running well. Those kinds of protections are not available in Linux or BSD AFAIK. For example you cannot spin the processor on System Z from an application for very long without being killed by the supervisor, but you can easily write userland code to hog the machine in *NIX. As *NIX was and is being changed to exploit SMP (and look at all the code that has been added let's say in the last 5 years to do this) it's very apparent to exploit the hardware threading is more useful than process forking. But that way of thinking is newish in *NIX and not all the tools and facilities (resource serialization etc) that are available in other systems (System Z for example) are available to *NIX so there are growing pains. A lot of progress has been made, no doubt. But there is still a lot of room for improvement. I can't say that in my experience pthread_x under 2.6 Linux is heavier or requires more resources than say, pthread or libthread under Solaris (though my experience is somewhat dated). Never the less, as I'm not sure we even agree on what the term SMP design means, lets agree not to agree... - Gilb ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On May 8, 2011, at 7:19 PM, guy keren wrote: when you say system Z - do you refer to what IBM formerly called MVS? IBM's had a lot of time to perfect it, their first multiprocessor machine was delivered in 1969. Geoff. -- Geoffrey S. Mendelson, N3OWJ/4X1GM Occam's Razor does not apply to electronics. If something won't turn on, it's not likely to be the power switch. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
OT: nokia n800 for sale
Hi list, I have a nokia n800. If any is interested I will sell it to the best bidder. The starting price for the device is 500 shekels. thanks, Meir ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
can't finish update: dpkg hangs installing xulrunner
apt-get update hangs at Setting up xulrunner-1.9.1 I can kill this, but then I can't finish the update because it says that dpkg was interrupted. Trying to let dpkg repair with sudo dpkg --configure -a hangs setting up xulrunner so I'm stuck. Any ideas? -- Michael Shiloh KA6RCQ www.teachmetomake.com teachmetomake.wordpress.com Interested in classes? Join http://groups.google.com/group/teach-me-to-make ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: can't finish update: dpkg hangs installing xulrunner
On Sun, 2011-05-08 at 17:04 -0700, Michael Shiloh wrote: apt-get update hangs at Setting up xulrunner-1.9.1 I can kill this, but then I can't finish the update because it says that dpkg was interrupted. Trying to let dpkg repair with sudo dpkg --configure -a hangs setting up xulrunner so I'm stuck. Any ideas? what does strace tell you? i.e. run the program under 'strace -f -o /tmp/somefile.txt dpkg ...' and when it hangs - look in the file /tmp/somefile.txt, and see what is it waiting for. --guy ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
On Sun, 2011-05-08 at 09:47 +0300, Nadav Har'El wrote: On Sat, May 07, 2011, Omer Zak wrote about Re: Disk I/O as a bottleneck?: I suspect that speeding up /usr won't help improve performance that much. The applications, which seem to be sluggish, deal with a lot of user data in /home. Furthermore, this user data varies a lot with time, hence it is not that good idea to store it in SSD. As usual, it depends on your workload applies. In my own personal experience (and naturally, it might differ considerably from your use case), when I see sluggish behavior on a desktop machine, what is actually happening is that foreground activity, such as playing or working with video files, or such as compilation of a large project, causes a lot of other pages to be swapped out; And then, when you switch to a different application, it needs to swap pages in - either program text (code) directly from the executables, or data pages from the swap partition. I explicitly checked whether my PC is swapping or not (vmstat 1) and it was not. The sluggish behavior was not due to any RAM shortage. The contents of /proc/sys/vm/swappiness are: 60 I did find (using iostat -x -k 1) that both /usr and /home are heavily used (and %util hits 100% for both of them). In my PC they are in separate partitions but on the same hard disk (which is managed by LVM with encryption, except for a smallish boot partition). [OFFTOPIC] It was great experience to see all this thread resulting from my single posting. I can understand how it comes that some people enjoy trolling mailing lists. /wicked_grin --- Omer -- We live in a world of stupid people who enjoy being stupid. My own blog is at http://www.zak.co.il/tddpirate/ My opinions, as expressed in this E-mail message, are mine alone. They do not represent the official policy of any organization with which I may be affiliated in any way. WARNING TO SPAMMERS: at http://www.zak.co.il/spamwarning.html ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Disk I/O as a bottleneck?
[This E-mail message is bottom-posting contrary to my usual custom.] On Sun, 2011-05-08 at 17:26 +0300, Gilboa Davara wrote: On Sat, 2011-05-07 at 15:29 +0300, Omer Zak wrote: I have a PC with powerful processor, lots of RAM and SATA hard disk. Nevertheless I noticed that sometimes applications (evolution E-mail software and Firefox[iceweasel] Web browser) have the sluggish feel of a busy system (command line response time remains crisp, however, because the processor is 4x2 core one [4 cores, each multithreads as 2]). I run the gnome-system-monitor all the time. I notice that even when those applications feel sluggish, only one or at most two CPUs have high utilization, and there is plenty of free RAM (no swap space is used at all). Disk I/O is not monitored by gnome-system-monitor. So I suspect that the system is slowed down by disk I/O. I would like to eliminate it as a possible cause for the applications' sluggish feel. I ran smartctl tests on the hard disk, and they gave it clean bill of health. Therefore I/O error recovery should not be the reason for performance degradation. I am asking Collective Wisdom for advice about how to do: 1. Monitoring disk I/O load (counting I/O requests is not sufficient, as each request takes different time to complete due for example to disk head seeks or platter rotation time). 2. Disk scheduler fine-tuning possibilities to optimize disk I/O handling. 3. If smartctl is not sufficient to ensure that no I/O error overhead is incurred, how to better assess the hard disk's health? Thanks, --- Omer Hello Omer, Two questions: 1. Kernel version? 2.6.38 added a very small patch that that done wonders to eliminate foreground process scheduling issues that plagued desktop setups since the introduction of the CFS scheduler*. (Google for +2.6.38 +group +scheduling) On my Fedora 14 + 2.6.38 / dual 6C Xeon workstation I can easily watching a DVD movies while compiling the kernel and running a couple of VM's (using qemu-kvm) in the background. Doing the same using the stock Fedora 14 2.6.35 kernel is... err... interesting experience :) Standard Debian Squeeze kernel: $ uname -a Linux c4 2.6.32-5-vserver-amd64 #1 SMP Mon Mar 7 23:14:47 UTC 2011 x86_64 GNU/Linux 2. Are you sure your SATA is in AHCI mode? (Simply type: $ find /sys/devices | grep ahci) My SATA indeed runs in non-AHCI mode. The question now is how to configure it to run in AHCI mode? According to the Wikipedia article http://en.wikipedia.org/wiki/Advanced_Host_Controller_Interface, I need to modify a BIOS setting and maybe create a new initrd image. Another source: http://www.techmetica.com/howto/sata-ahci-mode-bios-setting.-what-does-it-do/ I can handle the BIOS setting but how to check whether a new initrd image is needed, and if necessary create it? --- Omer -- Bottom posters are filthy heretics and infidels and ought to be burned on the stake after having been tarred, feathered and having rotten eggs thrown at their faces! My own blog is at http://www.zak.co.il/tddpirate/ My opinions, as expressed in this E-mail message, are mine alone. They do not represent the official policy of any organization with which I may be affiliated in any way. WARNING TO SPAMMERS: at http://www.zak.co.il/spamwarning.html ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: can't finish update: dpkg hangs installing xulrunner
On 05/08/2011 08:10 PM, guy keren wrote: On Sun, 2011-05-08 at 17:04 -0700, Michael Shiloh wrote: apt-get update hangs at Setting up xulrunner-1.9.1 I can kill this, but then I can't finish the update because it says that dpkg was interrupted. Trying to let dpkg repair with sudo dpkg --configure -a hangs setting up xulrunner so I'm stuck. Any ideas? what does strace tell you? i.e. run the program under 'strace -f -o /tmp/somefile.txt dpkg ...' and when it hangs - look in the file /tmp/somefile.txt, and see what is it waiting for. Good idea. I'm not quite sure how to understand the results. The file starts with: 10208 execve(/usr/bin/dpkg, [dpkg, --configure, -a], [/* 20 vars */]) = 0 and ends with: 10227 set_robust_list(0xb775e900, 0xc) = 0 10227 futex(0xbfd66510, FUTEX_WAKE_PRIVATE, 1) = 0 10227 futex(0xbfd66510, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, NULL, bfd66520) = -1 EAGAIN (Resource temporarily unavailable) 10227 rt_sigaction(SIGRTMIN, {0x3196e0, [], SA_SIGINFO}, NULL, 8) = 0 10227 rt_sigaction(SIGRT_1, {0x319760, [], SA_RESTART|SA_SIGINFO}, NULL, 8) = 0 10227 rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0 10227 getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0 10227 uname({sys=Linux, node=t60, ...}) = 0 10227 statfs64(/selinux, 84, {f_type=EXT2_SUPER_MAGIC, f_bsize=4096, f_blocks=27662569, f_bfree=5190843, f_bavail=3785658, f_files=14057472, f_ffree=13425663, f_fsid={-1774502679, 1875976236}, f_namelen=255, f_frsize=4096}) = 0 10227 open(/proc/cpuinfo, O_RDONLY) = 3 10227 read(3, processor\t: 0\nvendor_id\t: Genuin..., 1024) = 1024 10227 read(3, no\nfpu\t\t: yes\nfpu_exception\t: y..., 1024) = 382 10227 read(3, , 1024) = 0 10227 close(3) = 0 10227 readlink(/etc/malloc.conf, 0xbfd64fab, 4096) = -1 ENOENT (No such file or directory) 10227 mmap2(NULL, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb765d000 10227 futex(0xb775e890, FUTEX_WAIT_PRIVATE, 2, NULL and that's the last line. It's been stucke there for awhile and no new lines are added to the file, so it looks like that futex is what's blocking. But why and what that means I don't know. What do you think? ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: can't finish update: dpkg hangs installing xulrunner
try to look back at the file, and see if this futex (0xb775e890) was acquired earlier in the strace output, and not released. these futexes are used to implement pthread mutexes, and if a an application attempts to lock the same mutex twice - it will deadlock - and you'll see it blocked on the underlying futex. note: it's also possible that this futex call is used to implement a different synchronization mechanism, which is persistent, and if an application crashed while it held this lock - it could lead to a similar deadlock. note also that the PID of the process stuck on this futex is 10227, while the PID of the original dpkg process is 10208 - so dpkg launched a process which got stuck. looking back at the strace log file - you could find what command this process executes. --guy On Sun, 2011-05-08 at 21:31 -0700, Michael Shiloh wrote: On 05/08/2011 08:10 PM, guy keren wrote: On Sun, 2011-05-08 at 17:04 -0700, Michael Shiloh wrote: apt-get update hangs at Setting up xulrunner-1.9.1 I can kill this, but then I can't finish the update because it says that dpkg was interrupted. Trying to let dpkg repair with sudo dpkg --configure -a hangs setting up xulrunner so I'm stuck. Any ideas? what does strace tell you? i.e. run the program under 'strace -f -o /tmp/somefile.txt dpkg ...' and when it hangs - look in the file /tmp/somefile.txt, and see what is it waiting for. Good idea. I'm not quite sure how to understand the results. The file starts with: 10208 execve(/usr/bin/dpkg, [dpkg, --configure, -a], [/* 20 vars */]) = 0 and ends with: 10227 set_robust_list(0xb775e900, 0xc) = 0 10227 futex(0xbfd66510, FUTEX_WAKE_PRIVATE, 1) = 0 10227 futex(0xbfd66510, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, NULL, bfd66520) = -1 EAGAIN (Resource temporarily unavailable) 10227 rt_sigaction(SIGRTMIN, {0x3196e0, [], SA_SIGINFO}, NULL, 8) = 0 10227 rt_sigaction(SIGRT_1, {0x319760, [], SA_RESTART|SA_SIGINFO}, NULL, 8) = 0 10227 rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0 10227 getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0 10227 uname({sys=Linux, node=t60, ...}) = 0 10227 statfs64(/selinux, 84, {f_type=EXT2_SUPER_MAGIC, f_bsize=4096, f_blocks=27662569, f_bfree=5190843, f_bavail=3785658, f_files=14057472, f_ffree=13425663, f_fsid={-1774502679, 1875976236}, f_namelen=255, f_frsize=4096}) = 0 10227 open(/proc/cpuinfo, O_RDONLY) = 3 10227 read(3, processor\t: 0\nvendor_id\t: Genuin..., 1024) = 1024 10227 read(3, no\nfpu\t\t: yes\nfpu_exception\t: y..., 1024) = 382 10227 read(3, , 1024) = 0 10227 close(3) = 0 10227 readlink(/etc/malloc.conf, 0xbfd64fab, 4096) = -1 ENOENT (No such file or directory) 10227 mmap2(NULL, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb765d000 10227 futex(0xb775e890, FUTEX_WAIT_PRIVATE, 2, NULL and that's the last line. It's been stucke there for awhile and no new lines are added to the file, so it looks like that futex is what's blocking. But why and what that means I don't know. What do you think? ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il