[R] Speed up R
Robert McFadden writes: Dear R Users, I hope that there is someone who has an experience with a problem that I describe below and will help me. I must buy new desktop computer and I'm wondering which processor to choose if my only aim is to speed up R. I would like to reduce a simulation time - sometimes it takes days. I consider buying one of them (I'm working under Win XP 32 bit): 1. Intel Core2 Duo E6700 2.67 GHz 2. Dual-Core Intel Xeon processor 3070 - 2,66 GHz 3. AMD Athlon 64 X2 6000+ Or simple Pentium 4? I'm very confused because I'm not sure whether R takes advantage dual-core or not. If not, probably Athlon would be better, wouldn't be? I would appreciate any help. Rob Hi Robert, Let me suggest you a dirty solution - if simulations take days and you must run them many times I would have rewriten them, let say, in C. I had a program in Matlab which took more than an hour to run and I had to run it many times, so I usually prepared a few runs, started them in the evening before leaving the office and got the results next morning. After a while I have re-written it in C (this took me a few days) and got a spead-up factor of about 100, so that now the run took just a few minutes! Languages like R and Matlab are extreemely convenient but if performance is a very important issue you shoul use C, Fortran, C++, etc. Regards, Moshe Olshansky. __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Speed up R
Don't rush to buy new hardware yet (other than perhaps more RAM for your existing desktop). First of all you should make sure that your R code can't be made any faster. (I've seen cases where careful re-writes increased speed by a factor of 10 or more.) There are some rules (such as pre-allocate enough memory for vectors/lists, use matrices instead of data frames etc) and tools (?Rprof, ?Sys.time) that can help a lot. Check the manuals and the archives, for example http://article.gmane.org/gmane.comp.lang.r.general/48800 -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Robert McFadden Sent: Tuesday, June 19, 2007 4:51 PM To: r-help@stat.math.ethz.ch Subject: [R] Speed up R Dear R Users, I hope that there is someone who has an experience with a problem that I describe below and will help me. I must buy new desktop computer and I'm wondering which processor to choose if my only aim is to speed up R. I would like to reduce a simulation time - sometimes it takes days. I consider buying one of them (I'm working under Win XP 32 bit): 1. Intel Core2 Duo E6700 2.67 GHz 2. Dual-Core Intel Xeon processor 3070 - 2,66 GHz 3. AMD Athlon 64 X2 6000+ Or simple Pentium 4? I'm very confused because I'm not sure whether R takes advantage dual-core or not. If not, probably Athlon would be better, wouldn't be? I would appreciate any help. Rob __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Speed up R
Original Message Subject: Re:[R] Speed up R From: Matthew Keller [EMAIL PROTECTED] To: Robert McFadden [EMAIL PROTECTED] Date: 19.06.2007 23:09 but I think that windows has problems addressing that much RAM (surely the 64bit Vista is OK with it though... surely). Linux or Apple (the powermac) might be better bets if you're wanting to work with programs that use a lot of RAM. BTW, Intel does make 64 bit chips now. They use them in macs. The Core 2 Duo has 64bit processing and is not only used in Macs... The thing with the 64bit Vista is that there is much less software driver support at the moment. So if you will not use more than 4GB and want to use it for other purposes than you might prefer the 32bit Vista... My personal subjective feeling is that Vista is slower then XP. So I would say thats not the only reason to go for Linux. Stefan __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Speed up R
-Original Message- From: Prof Brian Ripley [mailto:[EMAIL PROTECTED] The advantage of dual processors is that you can use the machine for several things at once, including multiple R jobs. For example, when I am doing package checking I am typically checking 4 packages at once on a dual processor machine to get continuous high utilization. I would like to thank very much everybody taking part in discussion. Does an answer above suggest that I can open two R console and do simulations simultaneously? If so, all simulations take more or less 1/2 times - or much less then doing it in turn? During our discussion one mentioned that RAM is important. But in my computing I do not use up more then 500 MB. I have 786 MB it means (probably) that I have enough. Am I right? Best, Rob I have little doubt that a Pentium 4 would be much slower than the others. I've just bought an Intel Core 2 Duo E6600 primarily to run 64-bit Linux, but it also has Vista 64 and XP (32-bit) on it. I don't think the differences between the current dual-core chips are really enough to worry about: they will all look slow in less than a year. -- Brian D. Ripley, [EMAIL PROTECTED] Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/ University of Oxford, Tel: +44 1865 272861 (self) 1 South Parks Road, +44 1865 272866 (PA) Oxford OX1 3TG, UKFax: +44 1865 272595 __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Speed up R
Prof Brian Ripley [EMAIL PROTECTED] wrote: The advantage of dual processors is that you can use the machine for several things at once, including multiple R jobs. I've used dual-processor machines for about 10 years now and emphatically second the point made by Brian Ripley. Even if you are not running several R jobs at once, it is nice to do something else at full speed while a simulation uses another processor. Often overlooked is the additional large advantage to be gained by using SCSI disks rather than any form of ATA. With today's large applications and operating systems, there can be frequent paging, and a SCSI subsystem will make a large difference in computer response, at least under Windows. Unless your work is processor bound, disk I/O is usually what sets the pace. -- Mike Prager, NOAA, Beaufort, NC * Opinions expressed are personal and not represented otherwise. * Any use of tradenames does not constitute a NOAA endorsement. __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Speed up R
On Wed, 20 Jun 2007, Robert McFadden wrote: -Original Message- From: Prof Brian Ripley [mailto:[EMAIL PROTECTED] The advantage of dual processors is that you can use the machine for several things at once, including multiple R jobs. For example, when I am doing package checking I am typically checking 4 packages at once on a dual processor machine to get continuous high utilization. I would like to thank very much everybody taking part in discussion. Does an answer above suggest that I can open two R console and do simulations simultaneously? If so, all simulations take more or less 1/2 times - or much less then doing it in turn? Yes, you can. You will get very close to 2x speed up if you have enough (and fast enough) RAM. During our discussion one mentioned that RAM is important. But in my computing I do not use up more then 500 MB. I have 786 MB it means (probably) that I have enough. On a dual processor machine you need more to avoid any swapping. Even my 2.5 year old laptop has 1Gb, and I'd want at least 2Gb in a dual processor machine given that spec. My sysadmin suggests a minimum of 4Gb for 64-bit dual processors these days. Am I right? Best, Rob I have little doubt that a Pentium 4 would be much slower than the others. I've just bought an Intel Core 2 Duo E6600 primarily to run 64-bit Linux, but it also has Vista 64 and XP (32-bit) on it. I don't think the differences between the current dual-core chips are really enough to worry about: they will all look slow in less than a year. -- Brian D. Ripley, [EMAIL PROTECTED] Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/ University of Oxford, Tel: +44 1865 272861 (self) 1 South Parks Road, +44 1865 272866 (PA) Oxford OX1 3TG, UKFax: +44 1865 272595 __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- Brian D. Ripley, [EMAIL PROTECTED] Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/ University of Oxford, Tel: +44 1865 272861 (self) 1 South Parks Road, +44 1865 272866 (PA) Oxford OX1 3TG, UKFax: +44 1865 272595 __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Speed up R
Robert, I'm not exactly an expert, but here's what I think. If you have only 786 MB of RAM on your machine and you are using ~500 of it in a session of R, that could slow things down considerably because your machine is trying to find free blocks of memory that haven't been used yet. I would buy additional RAM. As for Mike Prager's point about the type of hard drive being important, I'm not sure this is right (someone correct me if I'm misunderstanding). R stores and accesses objects through RAM - they aren't stored and accessed on the hard drive except when reading and writing. So hard drive type probably won't make much difference to speed in R. Matt On 6/20/07, Robert McFadden [EMAIL PROTECTED] wrote: -Original Message- From: Prof Brian Ripley [mailto:[EMAIL PROTECTED] The advantage of dual processors is that you can use the machine for several things at once, including multiple R jobs. For example, when I am doing package checking I am typically checking 4 packages at once on a dual processor machine to get continuous high utilization. I would like to thank very much everybody taking part in discussion. Does an answer above suggest that I can open two R console and do simulations simultaneously? If so, all simulations take more or less 1/2 times - or much less then doing it in turn? During our discussion one mentioned that RAM is important. But in my computing I do not use up more then 500 MB. I have 786 MB it means (probably) that I have enough. Am I right? Best, Rob I have little doubt that a Pentium 4 would be much slower than the others. I've just bought an Intel Core 2 Duo E6600 primarily to run 64-bit Linux, but it also has Vista 64 and XP (32-bit) on it. I don't think the differences between the current dual-core chips are really enough to worry about: they will all look slow in less than a year. -- Brian D. Ripley, [EMAIL PROTECTED] Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/ University of Oxford, Tel: +44 1865 272861 (self) 1 South Parks Road, +44 1865 272866 (PA) Oxford OX1 3TG, UKFax: +44 1865 272595 __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- Matthew C Keller Postdoctoral Fellow Virginia Institute for Psychiatric and Behavioral Genetics __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Speed up R
FWIW, I have a dual processor G5 Mac (not dual core... 2 separate, 2GHz processors-- not even made anymore, so you might find a deal if trying to save $$. If not, go for the octo-core Mac Pro--- 2 quad cores in one very expensive box). On my system I have found that with 5G RAM installed, I can either 1) launch an R64 session that is truly 64 bit R environment, and address very large objects in RAM, or 2) do what has been described here and launch 2 sessions (in the command line, or batch sessions, or 1 GUI and 1 CLI session) and get nearly full speed out of both... a glance at the system activity monitor shows both processors pegged when I do this. But if I use one regular R session, I still have a very responsive system for other tasks. Good opportunity to use GnuCash to pay my bills I've never tried launching more than 2 sessions, since I only have the 2 cpus... On 6/19/07, Matthew Keller [EMAIL PROTECTED] wrote: Hi Robert, Here's my 2 cents. 64-bit is a memory issue, not a speed issue per se. If a concern is increasing RAM (which is important in R since objects are stored in RAM), then you will want to get 64 bit if you plan on getting a computer with over 4GB RAM. I'm not sure about this (someone correct me if I'm wrong), but I think that windows has problems addressing that much RAM (surely the 64bit Vista is OK with it though... surely). Linux or Apple (the powermac) might be better bets if you're wanting to work with programs that use a lot of RAM. BTW, Intel does make 64 bit chips now. They use them in macs. As for speed, go with multicore processors with as much GHz as possible. On 6/19/07, Robert McFadden [EMAIL PROTECTED] wrote: Dear R Users, I hope that there is someone who has an experience with a problem that I describe below and will help me. I must buy new desktop computer and I'm wondering which processor to choose if my only aim is to speed up R. I would like to reduce a simulation time - sometimes it takes days. I consider buying one of them (I'm working under Win XP 32 bit): 1. Intel Core2 Duo E6700 2.67 GHz 2. Dual-Core Intel Xeon processor 3070 - 2,66 GHz 3. AMD Athlon 64 X2 6000+ Or simple Pentium 4? I'm very confused because I'm not sure whether R takes advantage dual-core or not. If not, probably Athlon would be better, wouldn't be? I would appreciate any help. Rob __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- Matthew C Keller Postdoctoral Fellow Virginia Institute for Psychiatric and Behavioral Genetics __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- --- David L. Van Brunt, Ph.D. mailto:[EMAIL PROTECTED] If Tyranny and Oppression come to this land, it will be in the guise of fighting a foreign enemy. --James Madison [[alternative HTML version deleted]] __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Speed up R
Matthew Keller [EMAIL PROTECTED] wrote: Robert, ... As for Mike Prager's point about the type of hard drive being important, I'm not sure this is right (someone correct me if I'm misunderstanding). R stores and accesses objects through RAM - they aren't stored and accessed on the hard drive except when reading and writing. So hard drive type probably won't make much difference to speed in R. In my experience, it makes a substantial difference if any swapping to disk is going on. That will happen if, e.g., other processes or Windows itself need RAM. Though R keeps the data in RAM, under Windows, non-SCSI disk I/O puts a noticeable load on the CPU. As SCSI controllers have CPUs of their own, they offload much of that work from the system CPU. I have compared dual-processor computers with equal RAM, one with a SCSI subsystem and one with fast (7200 RPM) ATA disks and slightly faster CPUs. One was my work machine, one my home. The difference was not subtle. For another example, think of how slow laptops seem when multitasking, compared to a good workstation. It is usually the poor disk subsystem that's the bottleneck, not the CPU. Mike -- Mike Prager, NOAA, Beaufort, NC * Opinions expressed are personal and not represented otherwise. * Any use of tradenames does not constitute a NOAA endorsement. __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Speed up R
So Mike, let me ask you a question. If R runs out of RAM, does it begin to use virtual RAM, and hence begin to swap from the hard drive? If so, I could see how a faster hard drive would speed R up when you don't have enough RAM... On 6/20/07, Mike Prager [EMAIL PROTECTED] wrote: Matthew Keller [EMAIL PROTECTED] wrote: Robert, ... As for Mike Prager's point about the type of hard drive being important, I'm not sure this is right (someone correct me if I'm misunderstanding). R stores and accesses objects through RAM - they aren't stored and accessed on the hard drive except when reading and writing. So hard drive type probably won't make much difference to speed in R. In my experience, it makes a substantial difference if any swapping to disk is going on. That will happen if, e.g., other processes or Windows itself need RAM. Though R keeps the data in RAM, under Windows, non-SCSI disk I/O puts a noticeable load on the CPU. As SCSI controllers have CPUs of their own, they offload much of that work from the system CPU. I have compared dual-processor computers with equal RAM, one with a SCSI subsystem and one with fast (7200 RPM) ATA disks and slightly faster CPUs. One was my work machine, one my home. The difference was not subtle. For another example, think of how slow laptops seem when multitasking, compared to a good workstation. It is usually the poor disk subsystem that's the bottleneck, not the CPU. Mike -- Mike Prager, NOAA, Beaufort, NC * Opinions expressed are personal and not represented otherwise. * Any use of tradenames does not constitute a NOAA endorsement. __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- Matthew C Keller Postdoctoral Fellow Virginia Institute for Psychiatric and Behavioral Genetics __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Speed up R
Matthew Keller wrote: So Mike, let me ask you a question. If R runs out of RAM, does it begin to use virtual RAM, and hence begin to swap from the hard drive? If so, I could see how a faster hard drive would speed R up when you don't have enough RAM... Yes. Virtual memory management is done by any modern operating system. The slowdown will be extreme. (Therefore, a minimum of 2Gb is a good idea for serious crunching -- I'd recommend 3 or 4 if possible. Don't forget that any programming language may have two copies of some arrays in memory during certain operations.) But even when R itself is not using VM, any significant I/O load on a Windows CPU (when (S)ATA disks are used) slows down *at least* all other I/O, and it seems to me that it slows down other interrupt servicing (e.g., responding to mouse clicks) as well. Even if the latter is not strictly true, it may be that the mouse click requires paging something in, like the stupid animation that plays when files are copied. Aside: On a old PC, copying files from the command line was fine, but if I forgot did it from the Windows Explorer, the stupid animation swapped in from disk and the machine froze for ~30 seconds.) Windows Vista can take advantage of a new gizmo Intel has introducted with a 1 Gb solid-state disk cache. That might reduce such problems. Mike Mike Prager Southeast Fisheries Science Center, NOAA Beaufort, North Carolina USA __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] Speed up R
Dear R Users, I hope that there is someone who has an experience with a problem that I describe below and will help me. I must buy new desktop computer and I'm wondering which processor to choose if my only aim is to speed up R. I would like to reduce a simulation time - sometimes it takes days. I consider buying one of them (I'm working under Win XP 32 bit): 1. Intel Core2 Duo E6700 2.67 GHz 2. Dual-Core Intel Xeon processor 3070 - 2,66 GHz 3. AMD Athlon 64 X2 6000+ Or simple Pentium 4? I'm very confused because I'm not sure whether R takes advantage dual-core or not. If not, probably Athlon would be better, wouldn't be? I would appreciate any help. Rob __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Speed up R
Hi Robert, Here's my 2 cents. 64-bit is a memory issue, not a speed issue per se. If a concern is increasing RAM (which is important in R since objects are stored in RAM), then you will want to get 64 bit if you plan on getting a computer with over 4GB RAM. I'm not sure about this (someone correct me if I'm wrong), but I think that windows has problems addressing that much RAM (surely the 64bit Vista is OK with it though... surely). Linux or Apple (the powermac) might be better bets if you're wanting to work with programs that use a lot of RAM. BTW, Intel does make 64 bit chips now. They use them in macs. As for speed, go with multicore processors with as much GHz as possible. On 6/19/07, Robert McFadden [EMAIL PROTECTED] wrote: Dear R Users, I hope that there is someone who has an experience with a problem that I describe below and will help me. I must buy new desktop computer and I'm wondering which processor to choose if my only aim is to speed up R. I would like to reduce a simulation time - sometimes it takes days. I consider buying one of them (I'm working under Win XP 32 bit): 1. Intel Core2 Duo E6700 2.67 GHz 2. Dual-Core Intel Xeon processor 3070 - 2,66 GHz 3. AMD Athlon 64 X2 6000+ Or simple Pentium 4? I'm very confused because I'm not sure whether R takes advantage dual-core or not. If not, probably Athlon would be better, wouldn't be? I would appreciate any help. Rob __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- Matthew C Keller Postdoctoral Fellow Virginia Institute for Psychiatric and Behavioral Genetics __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Speed up R
On Tue, 19 Jun 2007, Robert McFadden wrote: Dear R Users, I hope that there is someone who has an experience with a problem that I describe below and will help me. I must buy new desktop computer and I'm wondering which processor to choose if my only aim is to speed up R. I would like to reduce a simulation time - sometimes it takes days. I consider buying one of them (I'm working under Win XP 32 bit): 1. Intel Core2 Duo E6700 2.67 GHz 2. Dual-Core Intel Xeon processor 3070 - 2,66 GHz 3. AMD Athlon 64 X2 6000+ Or simple Pentium 4? I'm very confused because I'm not sure whether R takes advantage dual-core or not. Not under Windows XP. The experience under Linux shows that on the right problem dual processors can help a lot (say 1.8x), but the gains are typically modest and can even be negative. The advantage of dual processors is that you can use the machine for several things at once, including multiple R jobs. For example, when I am doing package checking I am typically checking 4 packages at once on a dual processor machine to get continuous high utilization. If not, probably Athlon would be better, wouldn't be? I have little doubt that a Pentium 4 would be much slower than the others. I've just bought an Intel Core 2 Duo E6600 primarily to run 64-bit Linux, but it also has Vista 64 and XP (32-bit) on it. I don't think the differences between the current dual-core chips are really enough to worry about: they will all look slow in less than a year. -- Brian D. Ripley, [EMAIL PROTECTED] Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/ University of Oxford, Tel: +44 1865 272861 (self) 1 South Parks Road, +44 1865 272866 (PA) Oxford OX1 3TG, UKFax: +44 1865 272595 __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
