[R] Correlation question
I recently compared two different approaches to calculating the correlation of two variables, and I cannot explain the different results: data(cars) model - lm(dist~speed,data=cars) coef(model) fitted.right - model$fitted fitted.wrong - -17+5*cars$speed When using the OLS fitted values, the lines below all return the same R2 value: 1-sum((cars$dist-fitted.right)^2)/sum((cars$dist-mean(cars$dist))^2) cor(cars$dist,fitted.right)^2 (sum((cars$dist-mean(cars$dist))*(fitted.right-mean(fitted.right)))/(49*sd(cars$dist)*sd(fitted.right)))^2 However, when I use my estimated parameters to find the fitted values, fitted.wrong, the first equation returns a much lower R2 value, which I would expect since the fit is worse, but the other lines return the same R2 that I get when using the OLS fitted values. 1-sum((cars$dist-fitted.wrong)^2)/sum((cars$dist-mean(cars$dist))^2) cor(x=cars$dist,y=fitted.wrong)^2 (sum((cars$dist-mean(cars$dist))*(fitted.wrong-mean(fitted.wrong)))/(49*sd(cars$dist)*sd(fitted.wrong)))^2 I'm sure I'm missing something simple, but can someone explain the difference between these two methods of finding R2? Thanks. Jon [[alternative HTML version deleted]] __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Correlation question
Hi, try cor(fitted.right,fitted.wrong) should give 1 as both are a linear function of speed! Hence cor(cars$dist,fitted.right)^2 and cor(x=cars$dist,y=fitted.wrong)^2 must be the same. HTH d Feladó: R-help [r-help-boun...@r-project.org] ; meghatalmaz#243;: Jonathan Thayn [jth...@ilstu.edu] Küldve: 2015. február 21. 22:42 To: r-help@r-project.org Tárgy: [R] Correlation question I recently compared two different approaches to calculating the correlation of two variables, and I cannot explain the different results: data(cars) model - lm(dist~speed,data=cars) coef(model) fitted.right - model$fitted fitted.wrong - -17+5*cars$speed When using the OLS fitted values, the lines below all return the same R2 value: 1-sum((cars$dist-fitted.right)^2)/sum((cars$dist-mean(cars$dist))^2) cor(cars$dist,fitted.right)^2 (sum((cars$dist-mean(cars$dist))*(fitted.right-mean(fitted.right)))/(49*sd(cars$dist)*sd(fitted.right)))^2 However, when I use my estimated parameters to find the fitted values, fitted.wrong, the first equation returns a much lower R2 value, which I would expect since the fit is worse, but the other lines return the same R2 that I get when using the OLS fitted values. 1-sum((cars$dist-fitted.wrong)^2)/sum((cars$dist-mean(cars$dist))^2) cor(x=cars$dist,y=fitted.wrong)^2 (sum((cars$dist-mean(cars$dist))*(fitted.wrong-mean(fitted.wrong)))/(49*sd(cars$dist)*sd(fitted.wrong)))^2 I'm sure I'm missing something simple, but can someone explain the difference between these two methods of finding R2? Thanks. Jon [[alternative HTML version deleted]] __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] multiple parameter optimization with optim()
Harold,
Obviously the bottleneck is your objective function fn(). I have speeded up
your function by a factor of about 2.4 by using `outer' instead of sapply. I
think it can be speeded much more. I couldn't figure it out without spending a
lot of time. I am sure someone on this list-serv can come up with a cleverer
way to program the objective function.
pl3 - function(dat, Q, startVal = NULL, ...){
if(!is.null(startVal) length(startVal) != ncol(dat) ){
stop(Length of argument startVal not equal to
the number of parameters estimated)
}
if(!is.null(startVal)){
startVal - startVal
} else {
p - colMeans(dat)
startValA - rep(1, ncol(dat))
startValB - as.vector(log((1 - p)/p))
startVal - c(startValA,startValB)
}
rr1 - rr2 - numeric(nrow(dat))
qq - gauss.quad.prob(Q, dist = 'normal', mu = 0, sigma=1)
nds - qq$nodes
wts - qq$weights
Q - length(qq$nodes)
dat - as.matrix(dat)
fn - function(params){
a - params[1:20]
b - params[21:40]
for(j in 1:length(rr1)){
rr1[j] - sum(wts*exp(colSums(outer(dat[j,], nds,
function(x,y) dbinom(x, 1, 1/ (1 + exp(- 1.7 * a * (y - b))), log = TRUE)
}
-sum(log(rr1))
}
#opt - optim(startVal, fn, method = BFGS, hessian = TRUE)
opt - nlminb(startVal, fn, control=list(trace=1,
rel.tol=1.e-06, iter.max=50))
# opt - spg(startVal, fn)
opt
#list(coefficients = opt$par, LogLik = -opt$value,
Std.Error = sqrt(diag(solve(opt$hessian
}
Hope this is helpful,
Ravi
[[alternative HTML version deleted]]
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] Correlation question
Of course! Thank you, I knew I was missing something painfully obvious. Its seems, then, that this line 1-sum((cars$dist-fitted.wrong)^2)/sum((cars$dist-mean(cars$dist))^2) is finding something other than the traditional correlation. I found this in a lecture introducing correlation, but , now, I'm not sure what it is. It does do a better job of showing that the fitted.wrong variable is not a good prediction of the distance. On Feb 21, 2015, at 4:36 PM, Kehl Dániel wrote: Hi, try cor(fitted.right,fitted.wrong) should give 1 as both are a linear function of speed! Hence cor(cars$dist,fitted.right)^2 and cor(x=cars$dist,y=fitted.wrong)^2 must be the same. HTH d Feladó: R-help [r-help-boun...@r-project.org] ; meghatalmaz#243;: Jonathan Thayn [jth...@ilstu.edu] Küldve: 2015. február 21. 22:42 To: r-help@r-project.org Tárgy: [R] Correlation question I recently compared two different approaches to calculating the correlation of two variables, and I cannot explain the different results: data(cars) model - lm(dist~speed,data=cars) coef(model) fitted.right - model$fitted fitted.wrong - -17+5*cars$speed When using the OLS fitted values, the lines below all return the same R2 value: 1-sum((cars$dist-fitted.right)^2)/sum((cars$dist-mean(cars$dist))^2) cor(cars$dist,fitted.right)^2 (sum((cars$dist-mean(cars$dist))*(fitted.right-mean(fitted.right)))/(49*sd(cars$dist)*sd(fitted.right)))^2 However, when I use my estimated parameters to find the fitted values, fitted.wrong, the first equation returns a much lower R2 value, which I would expect since the fit is worse, but the other lines return the same R2 that I get when using the OLS fitted values. 1-sum((cars$dist-fitted.wrong)^2)/sum((cars$dist-mean(cars$dist))^2) cor(x=cars$dist,y=fitted.wrong)^2 (sum((cars$dist-mean(cars$dist))*(fitted.wrong-mean(fitted.wrong)))/(49*sd(cars$dist)*sd(fitted.wrong)))^2 I'm sure I'm missing something simple, but can someone explain the difference between these two methods of finding R2? Thanks. Jon [[alternative HTML version deleted]] __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Replacing 9999 and 999 values with NA
If you are reading the data frame using for instance read.csv, you can put in the argument na.string =. Another way to do that is data[data ==] - NA. It should be good to tell us how you are reading your dataset. On Feb 21, 2015 6:49 AM, Jeff Newmiller jdnew...@dcn.davis.ca.us wrote: You did not say how you imported the data, but if you used one of the read.table variants (including read.csv) then you can use the na.strings argument as documented in the help file for read.table. Next time please read the posting guide, as there are some useful tips in there, such as posting using plain text (a setting in your email program) so we don't get garbled info from you, and providing a reproducible example. --- Jeff NewmillerThe . . Go Live... DCN:jdnew...@dcn.davis.ca.usBasics: ##.#. ##.#. Live Go... Live: OO#.. Dead: OO#.. Playing Research Engineer (Solar/BatteriesO.O#. #.O#. with /Software/Embedded Controllers) .OO#. .OO#. rocks...1k --- Sent from my phone. Please excuse my brevity. On February 20, 2015 10:55:30 AM PST, Alexandra Catena amc5...@gmail.com wrote: Hello All, I have a data frame of two columns for wind. The first column is for wind speed and the second wind direction. I'm trying to replace the values in the first column and the 999 values in the second column with NA. I tried to use the function ltdl.fix.df but it doesn't seem to do anything. ltdl.fix.df(windMV, zero2na = FALSE, coded = 999) n = 9432 by p = 4 matrix checked, 0 NA(s) present 0 factor variable(s) present 5675 value(s) coded 999 set to NA 0 -ve value(s) set to +ve half the negative value I have R version 3.1.1 Thanks, Alexandra [[alternative HTML version deleted]] __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. [[alternative HTML version deleted]] __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] creating a distinct zip file
On 21 Feb 2015, at 18:01 , Erin Hodgess erinm.hodg...@gmail.com wrote: This is not for CRAN, just for someone else. It doesn't need to be submitted. As I understand it, it doesn't need to be submitted in order to be submitted... (to CRAN, winbuilder respectively). Just make sure it satisfies the formal package structure, with yourself in the maintainer field. Check out http://win-builder.r-project.org -pd Thanks, Erin On Sat, Feb 21, 2015 at 2:56 AM, Prof Brian Ripley rip...@stats.ox.ac.uk wrote: On 21/02/2015 07:31, Jeff Newmiller wrote: On Windows it builds a zip file. If you are on Linux, you might [1] need [2]. [1] https://stat.ethz.ch/pipermail/r-help/2005-January/063596.html [2] http://cran.r-project.org/doc/contributed/cross-build.pdf But the first is from 2005 and the second is invalid (it should be http://cran.r-project.org/doc/contrib/cross-build.pdf and describes R 1.7.x: what it describes is no longer supported). For some time you can install a package without compilable sources on any R platform. So the tarball would be all that is needed. If compilation is needed, submit to winbuilder to make .zip file. See also http://cran.r-project.org/bin/windows/base/rw-FAQ.html#How- can-I-get-a-binary-version-of-a-package_003f --- Jeff NewmillerThe . . Go Live... DCN:jdnew...@dcn.davis.ca.usBasics: ##.#. ##.#. Live Go... Live: OO#.. Dead: OO#.. Playing Research Engineer (Solar/BatteriesO.O#. #.O#. with /Software/Embedded Controllers) .OO#. .OO#. rocks...1k --- Sent from my phone. Please excuse my brevity. On February 20, 2015 6:56:34 PM PST, Rolf Turner r.tur...@auckland.ac.nz wrote: On 21/02/15 15:02, Jeff Newmiller wrote: R CMD INSTALL --build packagename That will create a *.tar.gz file, not a *.zip file. The latter being what Erin wanted, if I understand correctly. I have worked around the problem in the past with a shell script like unto: #! /bin/csh set vnum = `grep Version $pkge/DESCRIPTION | sed -e 's/Version: //'` R CMD INSTALL -l Lib $pkge /dev/null cd Lib zip -r -l $pkge.zip $pkge /dev/null mv $pkge.zip ../$pkge_$vnum.zip In the foregoing pkge is the name of the package you are trying to build. You will have to have created the holding library Lib a priori. There are doubtless (much) better ways of accomplishing this task, but I don't know them. cheers, Rolf --- Jeff NewmillerThe . . Go Live... DCN:jdnew...@dcn.davis.ca.usBasics: ##.#. ##.#. Live Go... Live: OO#.. Dead: OO#.. Playing Research Engineer (Solar/BatteriesO.O#. #.O#. with /Software/Embedded Controllers) .OO#. .OO#. rocks...1k --- Sent from my phone. Please excuse my brevity. On February 20, 2015 1:07:10 PM PST, Erin Hodgess erinm.hodg...@gmail.com wrote: Hello yet again. I am trying to create a zip file for a friend who has a Windows machine. He needs to access this via the local zip file packages option. When I use R CMD INSTALL --compile-both, it produces an item in the library tree (as promised). However, I would like to have an actual .zip file. I do know at one time that was possible, not sure if I can still do it. I did try R CMD INSTALL --force-biarch as well, same result as compile both. thank you for any suggestions. Sincerely, Erin __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/ posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- Brian D. Ripley, rip...@stats.ox.ac.uk Emeritus Professor of Applied Statistics, University of Oxford 1 South Parks Road, Oxford OX1 3TG, UK -- Erin Hodgess Associate Professor Department of Mathematical and Statistics University of Houston - Downtown mailto: erinm.hodg...@gmail.com [[alternative HTML version deleted]]
Re: [R] creating a distinct zip file
On Fri, Feb 20, 2015 at 6:56 PM, Rolf Turner r.tur...@auckland.ac.nz wrote: On 21/02/15 15:02, Jeff Newmiller wrote: R CMD INSTALL --build packagename That will create a *.tar.gz file, not a *.zip file. The latter being what Erin wanted, if I understand correctly. It depends on her system (I don't see it specified anywhere). On Windows, R CMD INSTALL --build packagename produces a compiled .zip file. On Mac it produces a .tgz. Haven't tried it on Linux. Peter __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] creating a distinct zip file
This is not for CRAN, just for someone else. It doesn't need to be submitted. Thanks, Erin On Sat, Feb 21, 2015 at 2:56 AM, Prof Brian Ripley rip...@stats.ox.ac.uk wrote: On 21/02/2015 07:31, Jeff Newmiller wrote: On Windows it builds a zip file. If you are on Linux, you might [1] need [2]. [1] https://stat.ethz.ch/pipermail/r-help/2005-January/063596.html [2] http://cran.r-project.org/doc/contributed/cross-build.pdf But the first is from 2005 and the second is invalid (it should be http://cran.r-project.org/doc/contrib/cross-build.pdf and describes R 1.7.x: what it describes is no longer supported). For some time you can install a package without compilable sources on any R platform. So the tarball would be all that is needed. If compilation is needed, submit to winbuilder to make .zip file. See also http://cran.r-project.org/bin/windows/base/rw-FAQ.html#How- can-I-get-a-binary-version-of-a-package_003f --- Jeff NewmillerThe . . Go Live... DCN:jdnew...@dcn.davis.ca.usBasics: ##.#. ##.#. Live Go... Live: OO#.. Dead: OO#.. Playing Research Engineer (Solar/BatteriesO.O#. #.O#. with /Software/Embedded Controllers) .OO#. .OO#. rocks...1k --- Sent from my phone. Please excuse my brevity. On February 20, 2015 6:56:34 PM PST, Rolf Turner r.tur...@auckland.ac.nz wrote: On 21/02/15 15:02, Jeff Newmiller wrote: R CMD INSTALL --build packagename That will create a *.tar.gz file, not a *.zip file. The latter being what Erin wanted, if I understand correctly. I have worked around the problem in the past with a shell script like unto: #! /bin/csh set vnum = `grep Version $pkge/DESCRIPTION | sed -e 's/Version: //'` R CMD INSTALL -l Lib $pkge /dev/null cd Lib zip -r -l $pkge.zip $pkge /dev/null mv $pkge.zip ../$pkge_$vnum.zip In the foregoing pkge is the name of the package you are trying to build. You will have to have created the holding library Lib a priori. There are doubtless (much) better ways of accomplishing this task, but I don't know them. cheers, Rolf --- Jeff NewmillerThe . . Go Live... DCN:jdnew...@dcn.davis.ca.usBasics: ##.#. ##.#. Live Go... Live: OO#.. Dead: OO#.. Playing Research Engineer (Solar/BatteriesO.O#. #.O#. with /Software/Embedded Controllers) .OO#. .OO#. rocks...1k --- Sent from my phone. Please excuse my brevity. On February 20, 2015 1:07:10 PM PST, Erin Hodgess erinm.hodg...@gmail.com wrote: Hello yet again. I am trying to create a zip file for a friend who has a Windows machine. He needs to access this via the local zip file packages option. When I use R CMD INSTALL --compile-both, it produces an item in the library tree (as promised). However, I would like to have an actual .zip file. I do know at one time that was possible, not sure if I can still do it. I did try R CMD INSTALL --force-biarch as well, same result as compile both. thank you for any suggestions. Sincerely, Erin __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/ posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- Brian D. Ripley, rip...@stats.ox.ac.uk Emeritus Professor of Applied Statistics, University of Oxford 1 South Parks Road, Oxford OX1 3TG, UK -- Erin Hodgess Associate Professor Department of Mathematical and Statistics University of Houston - Downtown mailto: erinm.hodg...@gmail.com [[alternative HTML version deleted]] __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] creating a distinct zip file
What is a compiled zip? There is a warning against using that term in section 1 of the Writing R Extensions documentation, along with a discussion of why it is ambiguous at best. Can you read that and clarify your statement? Since you make this assertion that R CMD INSTALL --build does not do what you want it to, I am not sure we are speaking of the same kind of file. If we are, then you may not have your system configured correctly. --- Jeff NewmillerThe . . Go Live... DCN:jdnew...@dcn.davis.ca.usBasics: ##.#. ##.#. Live Go... Live: OO#.. Dead: OO#.. Playing Research Engineer (Solar/BatteriesO.O#. #.O#. with /Software/Embedded Controllers) .OO#. .OO#. rocks...1k --- Sent from my phone. Please excuse my brevity. On February 21, 2015 11:47:16 AM PST, Erin Hodgess erinm.hodg...@gmail.com wrote: I have Windows, but the R CMD INSTALL --build pack does not produce a compiled zip. Thanks, Erin On Sat, Feb 21, 2015 at 1:49 PM, Peter Langfelder peter.langfel...@gmail.com wrote: On Fri, Feb 20, 2015 at 6:56 PM, Rolf Turner r.tur...@auckland.ac.nz wrote: On 21/02/15 15:02, Jeff Newmiller wrote: R CMD INSTALL --build packagename That will create a *.tar.gz file, not a *.zip file. The latter being what Erin wanted, if I understand correctly. It depends on her system (I don't see it specified anywhere). On Windows, R CMD INSTALL --build packagename produces a compiled .zip file. On Mac it produces a .tgz. Haven't tried it on Linux. Peter __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] creating a distinct zip file
I have Windows, but the R CMD INSTALL --build pack does not produce a compiled zip. Thanks, Erin On Sat, Feb 21, 2015 at 1:49 PM, Peter Langfelder peter.langfel...@gmail.com wrote: On Fri, Feb 20, 2015 at 6:56 PM, Rolf Turner r.tur...@auckland.ac.nz wrote: On 21/02/15 15:02, Jeff Newmiller wrote: R CMD INSTALL --build packagename That will create a *.tar.gz file, not a *.zip file. The latter being what Erin wanted, if I understand correctly. It depends on her system (I don't see it specified anywhere). On Windows, R CMD INSTALL --build packagename produces a compiled .zip file. On Mac it produces a .tgz. Haven't tried it on Linux. Peter -- Erin Hodgess Associate Professor Department of Mathematical and Statistics University of Houston - Downtown mailto: erinm.hodg...@gmail.com [[alternative HTML version deleted]] __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] creating a distinct zip file
On 21/02/2015 2:47 PM, Erin Hodgess wrote: I have Windows, but the R CMD INSTALL --build pack does not produce a compiled zip. What does it do? Can you show us a transcript? Duncan Murdoch Thanks, Erin On Sat, Feb 21, 2015 at 1:49 PM, Peter Langfelder peter.langfel...@gmail.com wrote: On Fri, Feb 20, 2015 at 6:56 PM, Rolf Turner r.tur...@auckland.ac.nz wrote: On 21/02/15 15:02, Jeff Newmiller wrote: R CMD INSTALL --build packagename That will create a *.tar.gz file, not a *.zip file. The latter being what Erin wanted, if I understand correctly. It depends on her system (I don't see it specified anywhere). On Windows, R CMD INSTALL --build packagename produces a compiled .zip file. On Mac it produces a .tgz. Haven't tried it on Linux. Peter __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
