Re: [R] updating observations in lm
Ivo: 1. You should not be fitting linear models as you describe. For why not and how they should be fit, consult a suitable text on numerical methods (e.g. Givens and Hoeting). 2. In R, I suggest using lm() and ?update, feeding update() data modified as you like. This is, after all, the reason for update(). -- Bert On Mon, May 27, 2013 at 8:12 AM, ivo welch ivo.we...@anderson.ucla.edu wrote: dear R experts---I would like to update OLS regressions with new observations on the front of the data, and delete some old observations from the rear. my goal is to have a flexible moving-window regression, with a minimum number of observations and a maximum number of observations. I can keep (X' X) and (X' y), and add or subtract observations from these two quantities myself, and then use crossprod. strucchange does recursive residuals, which is closely related, but it is not designed for such flexible movable windows, nor primarily designed to produce standard errors of coefficients. before I get started on this, I just wanted to inquire whether someone has already written such a function. regards, /iaw Ivo Welch (ivo.we...@gmail.com) __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- Bert Gunter Genentech Nonclinical Biostatistics Internal Contact Info: Phone: 467-7374 Website: http://pharmadevelopment.roche.com/index/pdb/pdb-functional-groups/pdb-biostatistics/pdb-ncb-home.htm __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] updating observations in lm
hi bert---thanks for the answer. my particular problem is well conditioned [stock returns] and speed is very important. about 4 years ago, I asked for speedier alternatives to lm (and you helped me on this one, too), and then checked into the speed/accuracy tradeoff. http://r.789695.n4.nabble.com/very-fast-OLS-regression-td884832.html . for the particular problem I had, solve(crossprod(x),crossprod(x,y)) worked reasonably well. moreover, it is easy to debug, being so simple. it was faster than lm() by a factor 5.. (for a more generic library use, it would be nice to have a warning flag when this algorithm fails, in which case it would fall back on a more robust algorithm or at least emit a warning. I wonder how much it would cost to check the condition of the matrix before deciding on the algorithm.) I looked at update(), but its documentation seems to refer to updating models, not observations. even if it did, given the speed of lm(), I don't think it will be that useful. regards, /iaw Ivo Welch (ivo.we...@gmail.com) On Mon, May 27, 2013 at 9:26 AM, Bert Gunter gunter.ber...@gene.com wrote: Ivo: 1. You should not be fitting linear models as you describe. For why not and how they should be fit, consult a suitable text on numerical methods (e.g. Givens and Hoeting). 2. In R, I suggest using lm() and ?update, feeding update() data modified as you like. This is, after all, the reason for update(). -- Bert On Mon, May 27, 2013 at 8:12 AM, ivo welch ivo.we...@anderson.ucla.edu wrote: dear R experts---I would like to update OLS regressions with new observations on the front of the data, and delete some old observations from the rear. my goal is to have a flexible moving-window regression, with a minimum number of observations and a maximum number of observations. I can keep (X' X) and (X' y), and add or subtract observations from these two quantities myself, and then use crossprod. strucchange does recursive residuals, which is closely related, but it is not designed for such flexible movable windows, nor primarily designed to produce standard errors of coefficients. before I get started on this, I just wanted to inquire whether someone has already written such a function. regards, /iaw Ivo Welch (ivo.we...@gmail.com) __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- Bert Gunter Genentech Nonclinical Biostatistics Internal Contact Info: Phone: 467-7374 Website: http://pharmadevelopment.roche.com/index/pdb/pdb-functional-groups/pdb-biostatistics/pdb-ncb-home.htm __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] updating observations in lm
?lm.fit ## may be useful to you then. Have you tried it? -- Bert On Mon, May 27, 2013 at 9:52 AM, ivo welch ivo.we...@gmail.com wrote: hi bert---thanks for the answer. my particular problem is well conditioned [stock returns] and speed is very important. about 4 years ago, I asked for speedier alternatives to lm (and you helped me on this one, too), and then checked into the speed/accuracy tradeoff. http://r.789695.n4.nabble.com/very-fast-OLS-regression-td884832.html . for the particular problem I had, solve(crossprod(x),crossprod(x,y)) worked reasonably well. moreover, it is easy to debug, being so simple. it was faster than lm() by a factor 5.. (for a more generic library use, it would be nice to have a warning flag when this algorithm fails, in which case it would fall back on a more robust algorithm or at least emit a warning. I wonder how much it would cost to check the condition of the matrix before deciding on the algorithm.) I looked at update(), but its documentation seems to refer to updating models, not observations. even if it did, given the speed of lm(), I don't think it will be that useful. regards, /iaw Ivo Welch (ivo.we...@gmail.com) On Mon, May 27, 2013 at 9:26 AM, Bert Gunter gunter.ber...@gene.com wrote: Ivo: 1. You should not be fitting linear models as you describe. For why not and how they should be fit, consult a suitable text on numerical methods (e.g. Givens and Hoeting). 2. In R, I suggest using lm() and ?update, feeding update() data modified as you like. This is, after all, the reason for update(). -- Bert On Mon, May 27, 2013 at 8:12 AM, ivo welch ivo.we...@anderson.ucla.edu wrote: dear R experts---I would like to update OLS regressions with new observations on the front of the data, and delete some old observations from the rear. my goal is to have a flexible moving-window regression, with a minimum number of observations and a maximum number of observations. I can keep (X' X) and (X' y), and add or subtract observations from these two quantities myself, and then use crossprod. strucchange does recursive residuals, which is closely related, but it is not designed for such flexible movable windows, nor primarily designed to produce standard errors of coefficients. before I get started on this, I just wanted to inquire whether someone has already written such a function. regards, /iaw Ivo Welch (ivo.we...@gmail.com) __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- Bert Gunter Genentech Nonclinical Biostatistics Internal Contact Info: Phone: 467-7374 Website: http://pharmadevelopment.roche.com/index/pdb/pdb-functional-groups/pdb-biostatistics/pdb-ncb-home.htm -- Bert Gunter Genentech Nonclinical Biostatistics Internal Contact Info: Phone: 467-7374 Website: http://pharmadevelopment.roche.com/index/pdb/pdb-functional-groups/pdb-biostatistics/pdb-ncb-home.htm __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] updating observations in lm
Look at the biglm package. It does 2 of the 3 things that you asked for: Construct an initial lm fit and add a new block of data to update that fit. It does not remove data, but you may be able to look at the code and figure out a way to modify it to do the final piece. On Mon, May 27, 2013 at 9:12 AM, ivo welch ivo.we...@anderson.ucla.eduwrote: dear R experts---I would like to update OLS regressions with new observations on the front of the data, and delete some old observations from the rear. my goal is to have a flexible moving-window regression, with a minimum number of observations and a maximum number of observations. I can keep (X' X) and (X' y), and add or subtract observations from these two quantities myself, and then use crossprod. strucchange does recursive residuals, which is closely related, but it is not designed for such flexible movable windows, nor primarily designed to produce standard errors of coefficients. before I get started on this, I just wanted to inquire whether someone has already written such a function. regards, /iaw Ivo Welch (ivo.we...@gmail.com) __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- Gregory (Greg) L. Snow Ph.D. 538...@gmail.com [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] updating observations in lm
The essential trick here is the Sherman-Morrison-Woodbury formula. My quantreg package has a lm.fit.recursive function that implements a fortran version for adding observations, but like biglm I don't remove observations at the other end either. Roger Koenker rkoen...@illinois.edu On May 27, 2013, at 2:07 PM, Greg Snow wrote: Look at the biglm package. It does 2 of the 3 things that you asked for: Construct an initial lm fit and add a new block of data to update that fit. It does not remove data, but you may be able to look at the code and figure out a way to modify it to do the final piece. On Mon, May 27, 2013 at 9:12 AM, ivo welch ivo.we...@anderson.ucla.eduwrote: dear R experts---I would like to update OLS regressions with new observations on the front of the data, and delete some old observations from the rear. my goal is to have a flexible moving-window regression, with a minimum number of observations and a maximum number of observations. I can keep (X' X) and (X' y), and add or subtract observations from these two quantities myself, and then use crossprod. strucchange does recursive residuals, which is closely related, but it is not designed for such flexible movable windows, nor primarily designed to produce standard errors of coefficients. before I get started on this, I just wanted to inquire whether someone has already written such a function. regards, /iaw Ivo Welch (ivo.we...@gmail.com) __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- Gregory (Greg) L. Snow Ph.D. 538...@gmail.com [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] updating observations in lm
On 27-05-2013, at 17:12, ivo welch ivo.we...@anderson.ucla.edu wrote: dear R experts---I would like to update OLS regressions with new observations on the front of the data, and delete some old observations from the rear. my goal is to have a flexible moving-window regression, with a minimum number of observations and a maximum number of observations. I can keep (X' X) and (X' y), and add or subtract observations from these two quantities myself, and then use crossprod. strucchange does recursive residuals, which is closely related, but it is not designed for such flexible movable windows, nor primarily designed to produce standard errors of coefficients. before I get started on this, I just wanted to inquire whether someone has already written such a function. For regression one would use a QR decomposition. There is an opensource Fortran library qrupdate (http://sourceforge.net/projects/qrupdate/) that can update an unpivoted QR decomposition for the case of deleting rows/columns and inserting rows/columns. It could be used to make an R package, which could be used for doing a moving window regression. Quite a lot of work. Berend __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] updating observations in lm
Gentlemans as 274 algorithm allows weights, so adding an obs with a weight of -1 would do the trick of removing obs, too. This may be a good job for hadwell wickhams c code interface. On May 27, 2013 12:47 PM, Berend Hasselman b...@xs4all.nl wrote: On 27-05-2013, at 17:12, ivo welch ivo.we...@anderson.ucla.edu wrote: dear R experts---I would like to update OLS regressions with new observations on the front of the data, and delete some old observations from the rear. my goal is to have a flexible moving-window regression, with a minimum number of observations and a maximum number of observations. I can keep (X' X) and (X' y), and add or subtract observations from these two quantities myself, and then use crossprod. strucchange does recursive residuals, which is closely related, but it is not designed for such flexible movable windows, nor primarily designed to produce standard errors of coefficients. before I get started on this, I just wanted to inquire whether someone has already written such a function. For regression one would use a QR decomposition. There is an opensource Fortran library qrupdate ( http://sourceforge.net/projects/qrupdate/) that can update an unpivoted QR decomposition for the case of deleting rows/columns and inserting rows/columns. It could be used to make an R package, which could be used for doing a moving window regression. Quite a lot of work. Berend [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] updating observations in lm
On 27-05-2013, at 21:57, ivo welch ivo.we...@gmail.com wrote: Gentlemans as 274 algorithm allows weights, so adding an obs with a weight of -1 would do the trick of removing obs, too. This may be a good job for hadwell wickhams c code interface. Searching for Gentlemans as 274 algorithm with google turned up this: http://jblevins.org/mirror/amiller/ where there is a fortran code for am updated AS 274 algorithm. I can't judge whether this is suitable for deleting observations. Berend __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
