Re: [R] speeding up a loop

2013-10-18 Thread Ye Lin
Thanks for your help David! I was running the same code the other day and it worked fine although it took a while as well. You are right that dff shud be df1 and maybe it's a portion of my data so it have an error of length =0. About CPU usage, I got it by clicking ctrl+alt+delete and it showed

Re: [R] speeding up a loop

2013-10-18 Thread jim holtman
You might want to use the profiler (Rprof) on a subset of your code to see where time is being spent. Find a subet that runs for a minute, or so, and enable profiling for the test. Take a look and see which functions are taking the time. This will be a start. You can also watch the task monitor

Re: [R] speeding up a loop

2013-10-18 Thread Ye Lin
Thanks for your advice Jim! I tried Rprof but since the code just freezes the system, I am not able to get results so far as I had to close R after waiting for a long time. I am confused that the same code would work differently on the same system. I tried out foreach package as well but didnt

Re: [R] speeding up a loop

2013-10-18 Thread jim holtman
When the system locks up, what do you see in the Task Manager? Is it consuming CPU and memory? On the example data you sent, you won't get a match on the time since there is not match for the first entry in df1 in the 'b' dataframe. This leads to an error that you are not checking for. Have

Re: [R] speeding up a loop

2013-10-18 Thread Ye Lin
when the system locks up it is consuming CPU to ~50%-60%, physical memory is about ~33%. I did tried on a subset of the data, e.g i in 1:100 and it works although takes about 5-10 min. Thanks for your help! On Fri, Oct 18, 2013 at 12:49 PM, jim holtman jholt...@gmail.com wrote: When the

[R] speeding up a loop

2013-10-17 Thread Ye Lin
Hey R professionals, I have a large dataset and I want to run a loop on it basically creating a new column which gathers information from another reference table. When I run the code, R just freezes and even does not response after 30min which is really unusual. I tried sapply as well but does

Re: [R] speeding up a loop

2013-10-17 Thread David Winsemius
On Oct 17, 2013, at 2:56 PM, Ye Lin wrote: Hey R professionals, I have a large dataset and I want to run a loop on it basically creating a new column which gathers information from another reference table. When I run the code, R just freezes and even does not response after 30min which

Re: [R] Speeding up a loop

2012-07-23 Thread wwreith
1.15 60 0.553555415 0.574892872 1.15 60 0.563183983 0.564029359 Shouldn't the function row out the second one, since it it higher in position 3 and lower in position 4 i.e. it should not all be yes? -- View this message in context:

Re: [R] Speeding up a loop

2012-07-23 Thread Rui Barradas
Hello, I think this is a boundary issue. In your op you've said less not less than or equal to. Try using = and = to see what happens, I bet it solves it. Rui Barradas Em 23-07-2012 14:43, wwreith escreveu: 1.15 60 0.553555415 0.574892872 1.15 60 0.563183983

Re: [R] Speeding up a loop

2012-07-21 Thread wwreith
next for loop question. I need a loop that removes a row from a matrix if it is worse in positions 1,2,3,4 than another row in the matrix. right now my matrix is 503028x26. Rule to define worse position1 is smaller, position2 is smaller, position3 is higher, and position4 is smaller Example:

Re: [R] Speeding up a loop

2012-07-21 Thread Rui Barradas
Hello, Maybe it would have been better to start a new thread, if the question is different. To show that it's a continuation, the subject line could be extended with part 2 or something like that. This solution runs in 3.6 hours. to.keep - function(x){ keep - function(i, env){

Re: [R] Speeding up a loop

2012-07-21 Thread wwreith
Any chance I could ask for an idiots guide for function to.keep(x). I understand how to use it but not what some of the lines are doing. Comments would be extremely helpful. -- View this message in context: http://r.789695.n4.nabble.com/Speeding-up-a-loop-tp4637201p4637316.html Sent from the R

Re: [R] Speeding up a loop

2012-07-21 Thread Rui Barradas
Ok, sorry, I should have included some comments. The function is divided in three parts, 1. intro, 2. decision, 3. keep rows. Part 3 is the function keep(), internal to to.keep(). Let's start with 1. 1. Setup some variables first. 1.a) The variables 'a'. If the input object 'x' is a matrix

[R] Speeding up a loop

2012-07-20 Thread wwreith
General problem: I have 20 projects that can be invested in and I need to decide which combinations meet a certain set of standards. The total possible combinations comes out to 2^20. However I know for a fact that the number of projects must be greater than 5 and less than 13. So far the the code

Re: [R] Speeding up a loop

2012-07-20 Thread Jean V Adams
I've had to do something similar, so I wrote a small function to help. This runs in about 1/4 the time of your code on my machine. Others may have a more efficient approach. all.combs - function(num, from=0, to=num) { # create a matrix of all possible combinations of num items #

Re: [R] Speeding up a loop

2012-07-20 Thread Petr Savicky
On Fri, Jul 20, 2012 at 05:45:30AM -0700, wwreith wrote: General problem: I have 20 projects that can be invested in and I need to decide which combinations meet a certain set of standards. The total possible combinations comes out to 2^20. However I know for a fact that the number of projects

Re: [R] Speeding up a loop

2012-07-20 Thread Jean V Adams
Petr, This is great. MUCH faster than the code I provided. And much more elegant code. Thanks for posting! Jean Petr Savicky savi...@cs.cas.cz wrote on 07/20/2012 09:26:34 AM: On Fri, Jul 20, 2012 at 05:45:30AM -0700, wwreith wrote: General problem: I have 20 projects that can be invested

Re: [R] Speeding up a loop

2012-07-20 Thread Petr Savicky
On Fri, Jul 20, 2012 at 04:26:34PM +0200, Petr Savicky wrote: On Fri, Jul 20, 2012 at 05:45:30AM -0700, wwreith wrote: General problem: I have 20 projects that can be invested in and I need to decide which combinations meet a certain set of standards. The total possible combinations comes

Re: [R] Speeding up a loop

2012-07-20 Thread Jean V Adams
Reith, William [USA] reith_will...@bah.com wrote on 07/20/2012 09:52:02 AM: Would this matrix eat up memory making the rest of my program slower? Each x needs to be multiplied by a matrix and the results checked against a set of thresholds. Doing them one at a time takes at least 24 hours

Re: [R] Speeding up a loop

2012-07-20 Thread wwreith
That is faster than what I was doing and reducing 15% of my iterations it still very helpful. Next question. I need to multiply each row x[i,] of the matrix x by another matrix A. Specifically for(i in 1:n) { If (x[i,]%*%A[,1].5 || x[i,]%*%A[,2]42 || x[i,]%*%A[,3]150) { x-x[-i,] n-n-1 }. #In

Re: [R] Speeding up a loop

2012-07-20 Thread Richard M. Heiberger
This works to multiply the ith row of a by the ith value of b. It might be what you can use a - matrix(1:30, 6, 5) b - 1:6 a a*b To simplify your code, I think you can do this---one multiplication xA - x %*% A Now you can do the tests on xA and not have any matrix multiplications inside the

Re: [R] Speeding up a loop

2012-07-20 Thread Richard M. Heiberger
whoops, backwards new.x - x[!remove.set,] On Fri, Jul 20, 2012 at 2:51 PM, Richard M. Heiberger r...@temple.eduwrote: This works to multiply the ith row of a by the ith value of b. It might be what you can use a - matrix(1:30, 6, 5) b - 1:6 a a*b To simplify your code, I think you