Re: [R] increase the usage of CPU and Memory
Hi Xi, Maybe you should try to parallelize your calculations. See package parallel. http://stat.ethz.ch/R-manual/R-devel/library/parallel/doc/parallel.pdf Arnaud On Mon, Jun 25, 2012 at 8:07 PM, Xi amzhan...@gmail.com wrote: Dear All, I have been searching online for help increasing my R code more efficiently for almost a whole day, however, there is no solution to my case. So if anyone could give any clue to solve my problem, I would be very appreciate for you help. Thanks in advance. Here is my issue: My desktop is with i7-950 Quad-core CPU with 24Gb memory, and a ?NVIDIA GTX 480 ?graphic card, and I am using a 64-bit version of R under 64-bit Windows . I am running a for loop to generate a 461*5 matrix data, which is coming from the coefficients of 5 models. The loop would produce 5 values one time, and it will run 461 times in total. I have tried to run the code inside the loop just once, it will cost almost 10 seconds, so if we intuitively calculate the time of the whole loop will cost, it would be 4610 seconds, equal to almost one and a half hours, which is exactly the whole loop taking indeed. But I have to run this kinda loop for 30 data-sets! Although I thought I am using a not-bad at all desktop, I checked the usage of CPU and memory during my running R code, and found out the whole code just used 15% of CPU and 10% of memory. Does anyone have the same issue with me? or Does anyone know some methods to shorten the running time and increase the usage of CPU and memory? Many thanks, Xi [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] increase the usage of CPU and Memory
Hello Xi, Have you tried replacing the for loop by an apply construct, e.g., lapply or sapply? In my experience these functions are more efficient than for. At any rate, if you succeed with, say, lapply, there are some R packages that support parallel processing versions. I believe the package parallel is now part of base R, and a version of multicore's mclapply (which, in multicore, does not support Windows) is available. Regards, Richard Richard R. Liu Hebelstr. 136 CH-4056 Basel Switzerland Tel.: +41 61 321 66 00 Mobil: +41 79 708 67 66 Email: richard@pueo-owl.ch __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] increase the usage of CPU and Memory
Dear All, I have been searching online for help increasing my R code more efficiently for almost a whole day, however, there is no solution to my case. So if anyone could give any clue to solve my problem, I would be very appreciate for you help. Thanks in advance. Here is my issue: My desktop is with i7-950 Quad-core CPU with 24Gb memory, and a NVIDIA GTX 480 graphic card, and I am using a 64-bit version of R under 64-bit Windows . I am running a for loop to generate a 461*5 matrix data, which is coming from the coefficients of 5 models. The loop would produce 5 values one time, and it will run 461 times in total. I have tried to run the code inside the loop just once, it will cost almost 10 seconds, so if we intuitively calculate the time of the whole loop will cost, it would be 4610 seconds, equal to almost one and a half hours, which is exactly the whole loop taking indeed. But I have to run this kinda loop for 30 data-sets! Although I thought I am using a not-bad at all desktop, I checked the usage of CPU and memory during my running R code, and found out the whole code just used 15% of CPU and 10% of memory. Does anyone have the same issue with me? or Does anyone know some methods to shorten the running time and increase the usage of CPU and memory? Many thanks, Xi [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] increase the usage of CPU and Memory
See the vignette for package 'parallel' to make use of your 4 cores. On 26/06/2012 01:07, Xi wrote: Dear All, I have been searching online for help increasing my R code more efficiently for almost a whole day, however, there is no solution to my case. So if anyone could give any clue to solve my problem, I would be very appreciate for you help. Thanks in advance. Here is my issue: My desktop is with i7-950 Quad-core CPU with 24Gb memory, and a NVIDIA GTX 480 graphic card, and I am using a 64-bit version of R under 64-bit Windows . I am running a for loop to generate a 461*5 matrix data, which is coming from the coefficients of 5 models. The loop would produce 5 values one time, and it will run 461 times in total. I have tried to run the code inside the loop just once, it will cost almost 10 seconds, so if we intuitively calculate the time of the whole loop will cost, it would be 4610 seconds, equal to almost one and a half hours, which is exactly the whole loop taking indeed. But I have to run this kinda loop for 30 data-sets! Although I thought I am using a not-bad at all desktop, I checked the usage of CPU and memory during my running R code, and found out the whole code just used 15% of CPU and 10% of memory. Does anyone have the same issue with me? or Does anyone know some methods to shorten the running time and increase the usage of CPU and memory? Many thanks, Xi [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- Brian D. Ripley, rip...@stats.ox.ac.uk Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/ University of Oxford, Tel: +44 1865 272861 (self) 1 South Parks Road, +44 1865 272866 (PA) Oxford OX1 3TG, UKFax: +44 1865 272595 __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] increase the usage of CPU and Memory
Hello Xi, If a program has input or output to disk or network, this may cause it to wait and not use the available CPU. Output is usually buffered, but may cause delay if the buffer gets full (I'm not sure though whether this is an issue with plenty of memory available) Take care Oliver On Mon, Jun 25, 2012 at 8:07 PM, Xi amzhan...@gmail.com wrote: Dear All, I have been searching online for help increasing my R code more efficiently for almost a whole day, however, there is no solution to my case. So if anyone could give any clue to solve my problem, I would be very appreciate for you help. Thanks in advance. Here is my issue: My desktop is with i7-950 Quad-core CPU with 24Gb memory, and a NVIDIA GTX 480 graphic card, and I am using a 64-bit version of R under 64-bit Windows . I am running a for loop to generate a 461*5 matrix data, which is coming from the coefficients of 5 models. The loop would produce 5 values one time, and it will run 461 times in total. I have tried to run the code inside the loop just once, it will cost almost 10 seconds, so if we intuitively calculate the time of the whole loop will cost, it would be 4610 seconds, equal to almost one and a half hours, which is exactly the whole loop taking indeed. But I have to run this kinda loop for 30 data-sets! Although I thought I am using a not-bad at all desktop, I checked the usage of CPU and memory during my running R code, and found out the whole code just used 15% of CPU and 10% of memory. Does anyone have the same issue with me? or Does anyone know some methods to shorten the running time and increase the usage of CPU and memory? Many thanks, Xi [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- Oliver Ruebenacker, Bioinformatics and Network Analysis Consultant President and Founder of Knowomics (http://www.knowomics.com/wiki/Oliver_Ruebenacker) Consultant at Predictive Medicine (http://predmed.com/people/oliverruebenacker.html) SBPAX: Turning Bio Knowledge into Math Models (http://www.sbpax.org) __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] increase the usage of CPU and Memory
On 26-06-2012 16:33, Oliver Ruebenacker wrote: Hello Xi, If a program has input or output to disk or network, this may cause it to wait and not use the available CPU. Output is usually buffered, but may cause delay if the buffer gets full (I'm not sure though whether this is an issue with plenty of memory available) Take care Oliver On Mon, Jun 25, 2012 at 8:07 PM, Xi amzhan...@gmail.com wrote: Dear All, I have been searching online for help increasing my R code more efficiently for almost a whole day, however, there is no solution to my case. So if anyone could give any clue to solve my problem, I would be very appreciate for you help. Thanks in advance. Here is my issue: My desktop is with i7-950 Quad-core CPU with 24Gb memory, and a NVIDIA GTX 480 graphic card, and I am using a 64-bit version of R under 64-bit Windows . I am running a for loop to generate a 461*5 matrix data, which is coming from the coefficients of 5 models. The loop would produce 5 values one time, and it will run 461 times in total. I have tried to run the code inside the loop just once, it will cost almost 10 seconds, so if we intuitively calculate the time of the whole loop will cost, it would be 4610 seconds, equal to almost one and a half hours, which is exactly the whole loop taking indeed. But I have to run this kinda loop for 30 data-sets! Although I thought I am using a not-bad at all desktop, I checked the usage of CPU and memory during my running R code, and found out the whole code just used 15% of CPU and 10% of memory. Does anyone have the same issue with me? or Does anyone know some methods to shorten the running time and increase the usage of CPU and memory? Many thanks, Xi [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. Hi Oliver, can you please give some details on what you are meaning by 'Output is usually buffered'? Thanks and regards, __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] increase the usage of CPU and Memory
Hello Christopher, If a process has data to write to hard disk, the data is usually written to a buffer in memory, and from there it is written to the hard disk independently of the CPU. Since writing to memory is much faster than writing to hard disk, this allows the process to run faster. To the process, it appears as if the data is already on disk. If, however, the buffer runs full, an attempt by a process to write more data will cause the process to wait until space is available in the buffer. If a process spends time waiting, it means it does not use all the CPU it could otherwise. I don't know how much input is buffered, but since only the process knows where it will request input from next, this limits ways to buffer input. I'm assuming though, that if you open a file and read the first few bytes, some more bytes may be read into a buffer since the process is likely to request them next. But in any case, input form disk or network is almost certain to cause waiting times and therefore decreases used CPU time. Take care Oliver On Tue, Jun 26, 2012 at 1:53 PM, Christofer Bogaso bogaso.christo...@gmail.com wrote: On 26-06-2012 16:33, Oliver Ruebenacker wrote: Hello Xi, If a program has input or output to disk or network, this may cause it to wait and not use the available CPU. Output is usually buffered, but may cause delay if the buffer gets full (I'm not sure though whether this is an issue with plenty of memory available) Take care Oliver On Mon, Jun 25, 2012 at 8:07 PM, Xi amzhan...@gmail.com wrote: Dear All, I have been searching online for help increasing my R code more efficiently for almost a whole day, however, there is no solution to my case. So if anyone could give any clue to solve my problem, I would be very appreciate for you help. Thanks in advance. Here is my issue: My desktop is with i7-950 Quad-core CPU with 24Gb memory, and a NVIDIA GTX 480 graphic card, and I am using a 64-bit version of R under 64-bit Windows . I am running a for loop to generate a 461*5 matrix data, which is coming from the coefficients of 5 models. The loop would produce 5 values one time, and it will run 461 times in total. I have tried to run the code inside the loop just once, it will cost almost 10 seconds, so if we intuitively calculate the time of the whole loop will cost, it would be 4610 seconds, equal to almost one and a half hours, which is exactly the whole loop taking indeed. But I have to run this kinda loop for 30 data-sets! Although I thought I am using a not-bad at all desktop, I checked the usage of CPU and memory during my running R code, and found out the whole code just used 15% of CPU and 10% of memory. Does anyone have the same issue with me? or Does anyone know some methods to shorten the running time and increase the usage of CPU and memory? Many thanks, Xi [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. Hi Oliver, can you please give some details on what you are meaning by 'Output is usually buffered'? Thanks and regards, -- Oliver Ruebenacker, Bioinformatics and Network Analysis Consultant President and Founder of Knowomics (http://www.knowomics.com/wiki/Oliver_Ruebenacker) Consultant at Predictive Medicine (http://predmed.com/people/oliverruebenacker.html) SBPAX: Turning Bio Knowledge into Math Models (http://www.sbpax.org) __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
