Re: [R] can we manage memory usage to increase speed?
On Mon, 1 Aug 2005, Haibo Huang wrote: Please refer to the following post. Which is about Windows only, not Linux. (And on Windows, the answer given is on the help page for memory.size. together with a better one.) Ed --- Mike Lawrence [EMAIL PROTECTED] wrote: Date: Mon, 1 Aug 2005 00:19:06 -0300 From: Mike Lawrence [EMAIL PROTECTED] To: Briggs, Meredith M [EMAIL PROTECTED] CC: r-help@stat.math.ethz.ch Subject: Re: [R] How do you increase memeory? memory.limit(size = x) where x is the desired memory limit in MB. Quoting Briggs, Meredith M [EMAIL PROTECTED]: Hello Function memory.size() =435109888. How do I increase it by, say 30%? Thanks Meredith --- Zhilin Liu [EMAIL PROTECTED] wrote: Hi, Thanks for reading. I am running a process in R for microarray data analysis. RedHat Enterprise Linux 4, dual AMD CPU, 6G memory. However, the R process use only a total of 200M memory. And the CPU usage is total to ~110% for two. The program takes at least 2 weeks to run at the current speed. Is there some way we can increase the usage of CPUs and memories and speed up? Any suggestion is appreciated. Thanks again. Zhilin [[alternative HTML version deleted]] __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html -- Brian D. Ripley, [EMAIL PROTECTED] Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/ University of Oxford, Tel: +44 1865 272861 (self) 1 South Parks Road, +44 1865 272866 (PA) Oxford OX1 3TG, UKFax: +44 1865 272595 __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Re: [R] can we manage memory usage to increase speed?
If you have a code that takes 2 weeks to run, than it might be a case of inefficient algorithm design. I was able to go from overnight runs (SELDI data analysis) to 20 minute runs by identifying single inefficient function that took most of the time, and writing it in C. Jarek \=== Jarek Tuszynski, PhD. o / \ Science Applications International Corporation \__,| (703) 676-4192 \ [EMAIL PROTECTED] `\ -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Zhilin Liu Sent: Monday, August 01, 2005 8:28 PM To: r-help@stat.math.ethz.ch Subject: [R] can we manage memory usage to increase speed? Hi, Thanks for reading. I am running a process in R for microarray data analysis. RedHat Enterprise Linux 4, dual AMD CPU, 6G memory. However, the R process use only a total of 200M memory. And the CPU usage is total to ~110% for two. The program takes at least 2 weeks to run at the current speed. Is there some way we can increase the usage of CPUs and memories and speed up? Any suggestion is appreciated. Thanks again. Zhilin [[alternative HTML version deleted]] __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Re: [R] can we manage memory usage to increase speed?
And you can identify inefficient code fairly easily taking snapshots from proc.time and computing elapsed time for sections of your code. spencer graves Tuszynski, Jaroslaw W. wrote: If you have a code that takes 2 weeks to run, than it might be a case of inefficient algorithm design. I was able to go from overnight runs (SELDI data analysis) to 20 minute runs by identifying single inefficient function that took most of the time, and writing it in C. Jarek \=== Jarek Tuszynski, PhD. o / \ Science Applications International Corporation \__,| (703) 676-4192 \ [EMAIL PROTECTED] `\ -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Zhilin Liu Sent: Monday, August 01, 2005 8:28 PM To: r-help@stat.math.ethz.ch Subject: [R] can we manage memory usage to increase speed? Hi, Thanks for reading. I am running a process in R for microarray data analysis. RedHat Enterprise Linux 4, dual AMD CPU, 6G memory. However, the R process use only a total of 200M memory. And the CPU usage is total to ~110% for two. The program takes at least 2 weeks to run at the current speed. Is there some way we can increase the usage of CPUs and memories and speed up? Any suggestion is appreciated. Thanks again. Zhilin [[alternative HTML version deleted]] __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html -- Spencer Graves, PhD Senior Development Engineer PDF Solutions, Inc. 333 West San Carlos Street Suite 700 San Jose, CA 95110, USA [EMAIL PROTECTED] www.pdf.com http://www.pdf.com Tel: 408-938-4420 Fax: 408-280-7915 __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Re: [R] can we manage memory usage to increase speed?
On 8/2/05, Spencer Graves [EMAIL PROTECTED] wrote: And you can identify inefficient code fairly easily taking snapshots from proc.time and computing elapsed time for sections of your code. spencer graves Using Rprof may be a better choice. See ?Rprof Tuszynski, Jaroslaw W. wrote: If you have a code that takes 2 weeks to run, than it might be a case of inefficient algorithm design. I was able to go from overnight runs (SELDI data analysis) to 20 minute runs by identifying single inefficient function that took most of the time, and writing it in C. Jarek \=== Jarek Tuszynski, PhD. o / \ Science Applications International Corporation \__,| (703) 676-4192 \ [EMAIL PROTECTED] `\ -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Zhilin Liu Sent: Monday, August 01, 2005 8:28 PM To: r-help@stat.math.ethz.ch Subject: [R] can we manage memory usage to increase speed? Hi, Thanks for reading. I am running a process in R for microarray data analysis. RedHat Enterprise Linux 4, dual AMD CPU, 6G memory. However, the R process use only a total of 200M memory. And the CPU usage is total to ~110% for two. The program takes at least 2 weeks to run at the current speed. Is there some way we can increase the usage of CPUs and memories and speed up? Any suggestion is appreciated. Thanks again. Zhilin [[alternative HTML version deleted]] __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html -- Spencer Graves, PhD Senior Development Engineer PDF Solutions, Inc. 333 West San Carlos Street Suite 700 San Jose, CA 95110, USA [EMAIL PROTECTED] www.pdf.com http://www.pdf.com Tel: 408-938-4420 Fax: 408-280-7915 __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Re: [R] can we manage memory usage to increase speed?
On Tue, 2 Aug 2005, Spencer Graves wrote: And you can identify inefficient code fairly easily taking snapshots from proc.time and computing elapsed time for sections of your code. Or use the profiler, which makes it much easier. There was a Programmers' Niche article about it in one of the first R Newsletters. -thomas spencer graves Tuszynski, Jaroslaw W. wrote: If you have a code that takes 2 weeks to run, than it might be a case of inefficient algorithm design. I was able to go from overnight runs (SELDI data analysis) to 20 minute runs by identifying single inefficient function that took most of the time, and writing it in C. Jarek \=== Jarek Tuszynski, PhD. o / \ Science Applications International Corporation \__,| (703) 676-4192 \ [EMAIL PROTECTED] `\ -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Zhilin Liu Sent: Monday, August 01, 2005 8:28 PM To: r-help@stat.math.ethz.ch Subject: [R] can we manage memory usage to increase speed? Hi, Thanks for reading. I am running a process in R for microarray data analysis. RedHat Enterprise Linux 4, dual AMD CPU, 6G memory. However, the R process use only a total of 200M memory. And the CPU usage is total to ~110% for two. The program takes at least 2 weeks to run at the current speed. Is there some way we can increase the usage of CPUs and memories and speed up? Any suggestion is appreciated. Thanks again. Zhilin [[alternative HTML version deleted]] __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html -- Spencer Graves, PhD Senior Development Engineer PDF Solutions, Inc. 333 West San Carlos Street Suite 700 San Jose, CA 95110, USA [EMAIL PROTECTED] www.pdf.com http://www.pdf.com Tel: 408-938-4420 Fax: 408-280-7915 __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html Thomas Lumley Assoc. Professor, Biostatistics [EMAIL PROTECTED] University of Washington, Seattle __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Re: [R] can we manage memory usage to increase speed?
Hi, Thank you all for the kind reply. I recompiled R as the previous one turned profiling off. I am using package MAANOVA, running the matest function which is a permutation test. The author did warn that it takes a long time to run. Here is one of the test results: [ Rdata]# ./R CMD Rprof maanovatest.out Each sample represents 0.02 seconds. Total run time: 905.95999466 seconds. Total seconds: time spent in function and callees. Self seconds: time spent in function alone. % total % self totalseconds selfsecondsname 100.00905.94 0.00 0.00 matest 80.18726.40 0.25 2.30 fitmaanova 79.37719.04 0.19 1.72 mixed 68.34619.16 1.05 9.50 pinv 64.33582.78 0.88 7.96 La.svd 55.51502.90 55.51502.90 .Call 38.47348.54 1.58 14.28 makeHq 34.50312.60 0.13 1.18 solveMME 19.80179.42 0.18 1.60 matest.engine 10.19 92.30 10.19 92.30 %*% .. The other part are not pasted as they are almost the same everytime we check the profiling. Only the parts above changes. For example, here is another output: 100.00 1411.02 0.00 0.02 matest 82.88 1169.40 0.24 3.40 fitmaanova 82.22 1160.18 0.19 2.74 mixed 68.82971.02 1.06 14.90 pinv 64.84914.94 0.85 11.98 La.svd 56.10791.64 56.10791.64 .Call 39.13552.10 1.55 21.84 makeHq 36.77518.82 0.13 1.88 solveMME 31.40443.04 0.00 0.00 matest.perm 17.10241.32 0.16 2.28 matest.engine 10.15143.24 10.15143.24 %*% I run this with a permutation of 2 times and it is still running. So it is not possible to run 1000 permutations with this kind of speed. And here is the output of TOP for R: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 15250 liuz 0 -20 218m 136m 2556 R 72.3 2.5 29:05.20 R Any suggestion to improve the performance is highly appreciated. Thanks a lot. Zhilin --- Douglas Bates [EMAIL PROTECTED] wrote: On 8/2/05, Spencer Graves [EMAIL PROTECTED] wrote: And you can identify inefficient code fairly easily taking snapshots from proc.time and computing elapsed time for sections of your code. spencer graves Using Rprof may be a better choice. See ?Rprof Tuszynski, Jaroslaw W. wrote: If you have a code that takes 2 weeks to run, than it might be a case of inefficient algorithm design. I was able to go from overnight runs (SELDI data analysis) to 20 minute runs by identifying single inefficient function that took most of the time, and writing it in C. Jarek \=== Jarek Tuszynski, PhD. o / \ Science Applications International Corporation \__,| (703) 676-4192 \ [EMAIL PROTECTED] `\ -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Zhilin Liu Sent: Monday, August 01, 2005 8:28 PM To: r-help@stat.math.ethz.ch Subject: [R] can we manage memory usage to increase speed? Hi, Thanks for reading. I am running a process in R for microarray data analysis. RedHat Enterprise Linux 4, dual AMD CPU, 6G memory. However, the R process use only a total of 200M memory. And the CPU usage is total to ~110% for two. The program takes at least 2 weeks to run at the current speed. Is there some way we can increase the usage of CPUs and memories and speed up? Any suggestion is appreciated. Thanks again. Zhilin [[alternative HTML version deleted]] __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html -- Spencer Graves, PhD Senior Development Engineer PDF Solutions, Inc. 333 West San Carlos Street Suite 700 San Jose, CA 95110, USA [EMAIL PROTECTED] www.pdf.com http://www.pdf.com Tel: 408-938-4420 Fax: 408-280-7915 __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting
Re: [R] can we manage memory usage to increase speed?
Please refer to the following post. Ed --- Mike Lawrence [EMAIL PROTECTED] wrote: Date: Mon, 1 Aug 2005 00:19:06 -0300 From: Mike Lawrence [EMAIL PROTECTED] To: Briggs, Meredith M [EMAIL PROTECTED] CC: r-help@stat.math.ethz.ch Subject: Re: [R] How do you increase memeory? memory.limit(size = x) where x is the desired memory limit in MB. Quoting Briggs, Meredith M [EMAIL PROTECTED]: Hello Function memory.size() =435109888. How do I increase it by, say 30%? Thanks Meredith [[alternative HTML version deleted]] __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html -- Mike Lawrence, BA(Hons) Research Assistant to Dr. Gail Eskes Dalhousie University QEII Health Sciences Centre (Psychiatry) [EMAIL PROTECTED] The road to Wisdom? Well, it's plain and simple to express: Err and err and err again, but less and less and less. - Piet Hein __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html --- Zhilin Liu [EMAIL PROTECTED] wrote: Hi, Thanks for reading. I am running a process in R for microarray data analysis. RedHat Enterprise Linux 4, dual AMD CPU, 6G memory. However, the R process use only a total of 200M memory. And the CPU usage is total to ~110% for two. The program takes at least 2 weeks to run at the current speed. Is there some way we can increase the usage of CPUs and memories and speed up? Any suggestion is appreciated. Thanks again. Zhilin [[alternative HTML version deleted]] __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
