On Fri, Dec 13, 2013 at 10:39:35AM -0500, Nick Holland wrote: > On 12/13/2013 09:10 AM, Zé Loff wrote: > >On Fri, Dec 13, 2013 at 02:44:26PM +0100, Peter Hessler wrote: > >>On 2013 Dec 13 (Fri) at 13:24:41 +0000 (+0000), Zé Loff wrote: > >>:On Fri, Dec 13, 2013 at 07:16:06AM -0600, Shawn K. Quinn wrote: > >>:> > >>:> I think R is using virtual memory as best it can, and I seriously doubt > >>:> you will get anything resembling satisfactory performance without > >>:> upgrading the RAM (memory) to 8Gb. > > > >[snip] > > > >>:> So: 1) segment this problem such that R never needs more than about 3Gb > >>:> of RAM in one run if possible, 2) upgrade the RAM, or 3) give R a very > >>:> long time to complete the task at hand and back up your hard disk > >>:> regularly because it will get a workout. > > > >[snip] > > > >>Using swap is a bug. Buy more ram. > > > >Thanks for your answers (and Marc's too, BTW). I never meant swapping to > >be more than a workaround, I wasn't expecting good performance. But I > >never expected it to render the machine virtually useless like it does, > >hence the first post. Off to the shop, then. > > > > swap is intended for things that are not currently being used much to be > pushed out of the way "for now" until they are needed again, presumably much > later (relatively speaking) > > It works great (relatively) when you have lots of stuff loaded and running > but are using only little parts at a time, when you can dump a big chunk of > unused RAM to disk, and bring in a big chunk of now desired data from disk > into RAM. > > That's not what you are doing. > > You have ONE application which is using huge amounts of data, that it is > thrashing all over. Odds are, if it was able to chunk the data up so it > could work on one little part, then another little part, then another little > part, 1) it would probably work great for you. 2) it would probably just do > this, keeping most of the data on disk, rather than sucking it all into RAM. > > If your app wants one "number" off something that is swapped out, it has to > bring in the whole swapped out page just to read or write that one value. > > You are running into the fact that memory is accessed on the order of > nanoseconds, and disk is accessed on the order of milliseconds, TIMES the > fact that any one location in RAM can be accessed almost as quickly as any > other location in RAM, but to get data swapped to disk requires a painfully > slow swap process of (relatively) huge blocks of data. > > you could be looking at million-to-one performance ratio here. Something > that could run in a minute in RAM might run for years in swap (that messes > up your upgrade plans :). > > Your application is a textbook example of "When swap fails". OpenBSD might > be able to manage its swap use better, but nothing will save you from what > you are trying to do. (well... ok, long, long ago... I've seen some > mainframes which, after you hit their physical RAM limits (16MB, iirc), > swapped to ... huge (for the day) RAM disks. But even then, the act of > swapping big pages of data out to get access to individal values of data > would be several orders of magnitude slower than a direct RAM access), > > Nick. >
Thanks for your time and for the detailed explanation (as you always do, which is great). I am well aware that swapping is crap (and why), especially in this particular case. I just didn't expect it to go so bad. The OP was just because I wasn't sure if I was doing things right. I saw the system practically freeze in a weird way (HD activity but frozen display, switching virtual consoles but no typing, etc), which was unexpected and made me wonder what the hell was going on. I've also run the same analysis on a mac also with 4Gb RAM running OS X (i.e. swapping), and it worked out nicely, so I thought it could be done here as well (and before the flames come out, *I know* that's comparing apples and oranges (no pun intended), and that there's little more than similar hardware between the two systems). I'll just get the right tool for the job: some more RAM. It'll make the RAM usage % indicator on my dwm desktop even more fun to look at. Cheers Zé --
