Re: [R] Vectorizing a for-loop for cross-validation in R

[Aleksandre Gavashelishvili]

Before posting on the r-help list I did run Rprof(). In my posting I asked
for help with re-writing the specific script into sapply() or
foreach()/doParallel format.

Thanks anyway for your time and suggestions,
Lexo

On Thu, Jan 24, 2019 at 12:38 AM Eric Berger  wrote:

> Charles writes about saving execution time by eliminating redundancies.
> If you see redundancies related to calling a time-consuming function
> multiple times with the same arguments, a very easy way to speed up your
> program is to memoise the functions using the package memoise.
>
> HTH,
> Eric
>
>
>
>
> On Wed, Jan 23, 2019 at 8:34 PM Berry, Charles  wrote:
>
>> See inline.
>>
>> > On Jan 23, 2019, at 2:17 AM, Aleksandre Gavashelishvili <
>> aleksandre.gavashelishv...@iliauni.edu.ge> wrote:
>> >
>> > I'm trying to speed up a script that otherwise takes days to handle
>> larger
>> > data sets. So, is there a way to completely vectorize or paralellize the
>> > following script:
>> >
>> >*# k-fold cross validation*
>> >
>> > df <- trees # a data frame 'trees' from R.
>> > df <- df[sample(nrow(df)), ] # randomly shuffles the data.
>> > k <- 10 # Number of folds. Note k=nrow(df) in the leave-one-out cross
>> > validation.
>> > folds <- cut(seq(from=1, to=nrow(df)), breaks=k, labels=FALSE) # creates
>> > unique numbers for k equally size folds.
>> > df$ID <- folds # adds fold IDs.
>> > df[paste("pred", 1:3, sep="")] <- NA # adds multiple columns "pred1"
>> > "pred2" "pred3" to speed up the following loop.
>> >
>> > library(mgcv)
>> >
>>
>> Rprof()
>>
>> replicate(100, {
>>
>>
>> > for(i in 1:k) {
>> >  # looping for different models:
>> >  m1 <- gam(Volume ~ s(Height), data=df, subset=(ID != i))
>> >  m2 <- gam(Volume ~ s(Girth), data=df, subset=(ID != i))
>> >  m3 <- gam(Volume ~ s(Girth) + s(Height), data=df, subset=(ID != i))
>> >
>> >  # looping for predictions:
>> >  df[df$ID==i, "pred1"] <- predict(m1, df[df$ID==i, ], type="response")
>> >  df[df$ID==i, "pred2"] <- predict(m2, df[df$ID==i, ], type="response")
>> >  df[df$ID==i, "pred3"] <- predict(m3, df[df$ID==i, ], type="response")
>> > }
>> >
>>
>> })
>>
>> Rprof(NULL)
>>
>> summaryRprof()
>>
>> ## read ?Rprof to get a sense of what it does
>>
>> ## read the summary to determine where time is being spent.
>>
>> ## the result was surprising to me. YMMV.
>>
>> ## there may be redundancies that you can eliminate by
>> ##  - doing the setup within gam() one time and saving it
>> ##  - calling the worker functions by modifying the setup
>> ##in a loop or function and saving the results
>>
>>
>> > # calculating residuals:
>> > df$res1 <- with(df, Volume - pred1)
>> > df$res2 <- with(df, Volume - pred2)
>> > df$res3 <- with(df, Volume - pred3)
>> >
>> > Model <- paste("m", 1:3, sep="") # creates a vector of model names.
>> >
>> > # creating a vector of mean-square errors (MSE):
>> > MSE <- with(df, c(
>> >  sum(res1^2) / nrow(df),
>> >  sum(res2^2) / nrow(df),
>> >  sum(res3^2) / nrow(df)
>> > ))
>> >
>> > model.mse <- data.frame(Model, MSE) # creates a data frame of model
>> names
>> > and mean-square errors.
>> > model.mse <- model.mse[order(model.mse$MSE), ] # rearranges the previous
>> > data frame in order of increasing mean-square errors.
>> >
>> > I'd appreciate any help. This code takes several days if run on >=30,000
>> > different GAM models and 3 predictors. Could you please help with
>> > re-writing the script into sapply() or foreach()/doParallel format?
>> >
>>
>> This is something you should learn to do. It is pretty standard practice.
>> Use the body of your for loop as the body of a function, add arguments, and
>> create a suitable return value. The something like
>>
>> lapply( 1:k, your.loop.body.function, other.arg1, other.arg2, ...)
>>
>> should work.  If it does, then parallel::mclapply(...) should also work.
>>
>> HTH,
>>
>> Chuck
>>
>>
>> __
>> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>>
>

[[alternative HTML version deleted]]

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] Unable to compute Confidence Intervals from output from MARSS package

[Ashim Kapoor]

Dear Bill,

Appreciate all your effort. I hope some one here can respond to this query.

Many thanks,
Ashim




On Thu, Jan 24, 2019 at 12:49 AM Bill Poling  wrote:

>
> Ashim.
>
> I see where I was mistaken, using MARSSparamsCIs(fit) <--Somehow I got an
> s in between param & Cis.
>
> I now get new error similarly as you, my apologies.
>
> final <- MARSSparamCIs(fit)
> Error in dpari[time.varying] <- dparmat(MLEobj, time.varying, t = t) :
>  replacement has length zero
>
> WHP
>
> From: R-help  On Behalf Of Ashim Kapoor
> Sent: Wednesday, January 23, 2019 1:38 AM
> To: r-help@r-project.org
> Subject: [R] Unable to compute Confidence Intervals from output from MARSS
> package
>
> Dear All,
>
> I am trying to use this package --->
> https://cran.r-project.org/web/packages/MARSS/index.html
> I am reading this book which shows some examples based on the above package
> ---> https://nwfsc-timeseries.github.io/atsa-labs/
>
> In a few words, the incantation MARSS(...) estimates the parameters and
> MARSSparamCIs should return the confidence intervals. My problem is that
> MARSSparamCIs returns this error :-
>
> > MARSSparamCIs(fit)
> Error in MARSSharveyobsFI(MLEobj) : replacement has length zero
> >
>
> ### This page and the next page, explains the jargon used :
>
> https://nwfsc-timeseries.github.io/atsa-labs/sec-dlm-example-of-a-univariate-dlm.html
>
> ### Here is a MWE to recreate the error :-
>
> # This is a Time Varying Parameters regression
> # Here a is fixed and b is doing a RW.
>
> x1 <- rnorm(1000)
> b <- cumsum(rnorm(1000, sd = sqrt(.6)))
> y <- 1 + b*x1 + rnorm(1000)
>
> Z = array(NA,c(1,2,1000))
> Z[1,1,] = rep(1,1000)
> Z[1,2,] = x1
>
> mod2.list = list ( B = matrix(c(1,0,0,1),nrow = 2 , byrow = T), U =
> matrix(0,2,1),
> Q = matrix(list(0,0,0,"s2b"),2,2), Z= Z, A = matrix(0), R = matrix("r"))
>
> fit = MARSS(as.vector(y), model = mod2.list,inits =
> list(x0=matrix(0,nrow=2,ncol=1)))
>
> MARSSparamsCIs(fit)
>
> # The above will give this error :
> Error in MARSSharveyobsFI(MLEobj) : replacement has length zero
>
> Best Regards,
> Ashim
>
> [[alternative HTML version deleted]]
>
> __
> mailto:R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
> Confidentiality Notice This message is sent from Zelis. This transmission
> may contain information which is privileged and confidential and is
> intended for the personal and confidential use of the named recipient only.
> Such information may be protected by applicable State and Federal laws from
> this disclosure or unauthorized use. If the reader of this message is not
> the intended recipient, or the employee or agent responsible for delivering
> the message to the intended recipient, you are hereby notified that any
> disclosure, review, discussion, copying, or taking any action in reliance
> on the contents of this transmission is strictly prohibited. If you have
> received this transmission in error, please contact the sender immediately.
> Zelis, 2018.
>
>
>

[[alternative HTML version deleted]]

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] Error "sufficient values in manual scale. 10 needed but only 7 provided"

[greg holly]

Hi Jeff;

I figured out the problem. I do apologize to you and members in the list to
bother you with this simple problem.

Regards,
Greg

On Wed, Jan 23, 2019 at 6:46 PM Jeff Newmiller 
wrote:

> Problem is in your data not matching your values, but you did not share
> your data. Try using the unique() function to see what values you have in
> your data.
>
> I will say that when I want to assign discrete colors I always start by
> converting my character column in the data frame to a factor and specify
> the order I want to see the levels presented in that conversion step. Then
> the colors only need to be specified in that same order and I don't need to
> keep repeating the labels in multiple places.
>
> library(ggplot2)
>
> mpg$classf <- factor( mpg$class
>, levels = c( "2seater"
>, "subcompact"
>, "compact"
>, "midsize"
>, "minivan"
>, "suv"
>, "pickup"
>)
>)
>
> class_colours <- rainbow( length( levels( mpg$classf ) ) )
> ggplot( mpg, aes( x = classf, y = cty, colour=classf ) ) +
>  geom_boxplot() +
>  scale_colour_manual( name="Class", values = class_colours )
>
>
>
>
> On January 23, 2019 3:12:26 PM PST, greg holly 
> wrote:
> >Hi Dear all;
> >
> >I am getting the "sufficient values in manual scale. 10 needed but only
> >7
> >provided." problem when running the followings. Your help is highly
> >appreciated.
> >
> >Regards,
> >Greg
> >
> >p2<-p1+scale_color_manual(name="Diseases",
> >labels=c("Myocardial Infarction", "Coronary artery disease", "Stroke",
> >"Hypertension", "Depression Anxiety Emotional Problems",  "Circulatory
> >Problems", "Diabetes"),
> >
> >values=c("Myocardial Infarction"="red",  "Coronary artery
> >disease"="purple","Stroke"="darkgreen",
> > "Hypertension"="orange",  "Depression Anxiety Emotional
> >Problems"="darkblue",
> > "Circulatory  Problems"="darkred","Diabetes"="blue"))
> >
> >   [[alternative HTML version deleted]]
> >
> >__
> >R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> >https://stat.ethz.ch/mailman/listinfo/r-help
> >PLEASE do read the posting guide
> >http://www.R-project.org/posting-guide.html
> >and provide commented, minimal, self-contained, reproducible code.
>
> --
> Sent from my phone. Please excuse my brevity.
>

[[alternative HTML version deleted]]

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] Error "sufficient values in manual scale. 10 needed but only 7 provided"

[Jeff Newmiller]

Problem is in your data not matching your values, but you did not share your 
data. Try using the unique() function to see what values you have in your data.

I will say that when I want to assign discrete colors I always start by 
converting my character column in the data frame to a factor and specify the 
order I want to see the levels presented in that conversion step. Then the 
colors only need to be specified in that same order and I don't need to keep 
repeating the labels in multiple places.

library(ggplot2)

mpg$classf <- factor( mpg$class
   , levels = c( "2seater"
   , "subcompact"
   , "compact"
   , "midsize"
   , "minivan"
   , "suv"
   , "pickup" 
   ) 
   )

class_colours <- rainbow( length( levels( mpg$classf ) ) )
ggplot( mpg, aes( x = classf, y = cty, colour=classf ) ) +
 geom_boxplot() +
 scale_colour_manual( name="Class", values = class_colours )




On January 23, 2019 3:12:26 PM PST, greg holly  wrote:
>Hi Dear all;
>
>I am getting the "sufficient values in manual scale. 10 needed but only
>7
>provided." problem when running the followings. Your help is highly
>appreciated.
>
>Regards,
>Greg
>
>p2<-p1+scale_color_manual(name="Diseases",
>labels=c("Myocardial Infarction", "Coronary artery disease", "Stroke",
>"Hypertension", "Depression Anxiety Emotional Problems",  "Circulatory
>Problems", "Diabetes"),
>
>values=c("Myocardial Infarction"="red",  "Coronary artery
>disease"="purple","Stroke"="darkgreen",
> "Hypertension"="orange",  "Depression Anxiety Emotional
>Problems"="darkblue",
> "Circulatory  Problems"="darkred","Diabetes"="blue"))
>
>   [[alternative HTML version deleted]]
>
>__
>R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
>https://stat.ethz.ch/mailman/listinfo/r-help
>PLEASE do read the posting guide
>http://www.R-project.org/posting-guide.html
>and provide commented, minimal, self-contained, reproducible code.

-- 
Sent from my phone. Please excuse my brevity.

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

[R] Error "sufficient values in manual scale. 10 needed but only 7 provided"

[greg holly]

Hi Dear all;

I am getting the "sufficient values in manual scale. 10 needed but only 7
provided." problem when running the followings. Your help is highly
appreciated.

Regards,
Greg

p2<-p1+scale_color_manual(name="Diseases",
labels=c("Myocardial Infarction", "Coronary artery disease", "Stroke",
"Hypertension", "Depression Anxiety Emotional Problems",  "Circulatory
Problems", "Diabetes"),

values=c("Myocardial Infarction"="red",  "Coronary artery
disease"="purple","Stroke"="darkgreen",
 "Hypertension"="orange",  "Depression Anxiety Emotional
Problems"="darkblue",
 "Circulatory  Problems"="darkred","Diabetes"="blue"))

[[alternative HTML version deleted]]

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] read_xl question

[Ista Zahn]

Something like

files <- list.files(pattern="*.xls", full.names = TRUE)
data <- lapply(files, read_excel, sheet="Flow Data", range=("b9:c10"))

should do it.

--Ista

On Wed, Jan 23, 2019 at 12:42 PM Thomas Subia via R-help
 wrote:
>
>
> Colleagues,
>
>  I have a workbook which has 3 worksheets
>
> I need to extract data from two specific cells from one ofthose worksheets.
>
>
>
> I can use read_excel to do this for one file.
>
> data<-read_excel("C:/Desktop/Excel_raw_data/0020-49785 8768.xls",
>
> sheet="Flow Data",range=("b9:c10"))
>
>
>
> How can I do this for all my Excel files in the directory?
>
>
>
> I can get the list of Excel files using: files =list.files(pattern="*.xls")
>
> But I’m not sure where to go from here.
>
> Some guidance would be appreciated.
>
>
>
> All the best
>
>  Thomas Subia
>
> Thomas Subia
>
> [[alternative HTML version deleted]]
>
> __
> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] Vectorizing a for-loop for cross-validation in R

[Eric Berger]

Charles writes about saving execution time by eliminating redundancies.
If you see redundancies related to calling a time-consuming function
multiple times with the same arguments, a very easy way to speed up your
program is to memoise the functions using the package memoise.

HTH,
Eric




On Wed, Jan 23, 2019 at 8:34 PM Berry, Charles  wrote:

> See inline.
>
> > On Jan 23, 2019, at 2:17 AM, Aleksandre Gavashelishvili <
> aleksandre.gavashelishv...@iliauni.edu.ge> wrote:
> >
> > I'm trying to speed up a script that otherwise takes days to handle
> larger
> > data sets. So, is there a way to completely vectorize or paralellize the
> > following script:
> >
> >*# k-fold cross validation*
> >
> > df <- trees # a data frame 'trees' from R.
> > df <- df[sample(nrow(df)), ] # randomly shuffles the data.
> > k <- 10 # Number of folds. Note k=nrow(df) in the leave-one-out cross
> > validation.
> > folds <- cut(seq(from=1, to=nrow(df)), breaks=k, labels=FALSE) # creates
> > unique numbers for k equally size folds.
> > df$ID <- folds # adds fold IDs.
> > df[paste("pred", 1:3, sep="")] <- NA # adds multiple columns "pred1"
> > "pred2" "pred3" to speed up the following loop.
> >
> > library(mgcv)
> >
>
> Rprof()
>
> replicate(100, {
>
>
> > for(i in 1:k) {
> >  # looping for different models:
> >  m1 <- gam(Volume ~ s(Height), data=df, subset=(ID != i))
> >  m2 <- gam(Volume ~ s(Girth), data=df, subset=(ID != i))
> >  m3 <- gam(Volume ~ s(Girth) + s(Height), data=df, subset=(ID != i))
> >
> >  # looping for predictions:
> >  df[df$ID==i, "pred1"] <- predict(m1, df[df$ID==i, ], type="response")
> >  df[df$ID==i, "pred2"] <- predict(m2, df[df$ID==i, ], type="response")
> >  df[df$ID==i, "pred3"] <- predict(m3, df[df$ID==i, ], type="response")
> > }
> >
>
> })
>
> Rprof(NULL)
>
> summaryRprof()
>
> ## read ?Rprof to get a sense of what it does
>
> ## read the summary to determine where time is being spent.
>
> ## the result was surprising to me. YMMV.
>
> ## there may be redundancies that you can eliminate by
> ##  - doing the setup within gam() one time and saving it
> ##  - calling the worker functions by modifying the setup
> ##in a loop or function and saving the results
>
>
> > # calculating residuals:
> > df$res1 <- with(df, Volume - pred1)
> > df$res2 <- with(df, Volume - pred2)
> > df$res3 <- with(df, Volume - pred3)
> >
> > Model <- paste("m", 1:3, sep="") # creates a vector of model names.
> >
> > # creating a vector of mean-square errors (MSE):
> > MSE <- with(df, c(
> >  sum(res1^2) / nrow(df),
> >  sum(res2^2) / nrow(df),
> >  sum(res3^2) / nrow(df)
> > ))
> >
> > model.mse <- data.frame(Model, MSE) # creates a data frame of model names
> > and mean-square errors.
> > model.mse <- model.mse[order(model.mse$MSE), ] # rearranges the previous
> > data frame in order of increasing mean-square errors.
> >
> > I'd appreciate any help. This code takes several days if run on >=30,000
> > different GAM models and 3 predictors. Could you please help with
> > re-writing the script into sapply() or foreach()/doParallel format?
> >
>
> This is something you should learn to do. It is pretty standard practice.
> Use the body of your for loop as the body of a function, add arguments, and
> create a suitable return value. The something like
>
> lapply( 1:k, your.loop.body.function, other.arg1, other.arg2, ...)
>
> should work.  If it does, then parallel::mclapply(...) should also work.
>
> HTH,
>
> Chuck
>
>
> __
> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

[[alternative HTML version deleted]]

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] Unable to compute Confidence Intervals from output from MARSS package

[Bill Poling]

Ashim.

I see where I was mistaken, using MARSSparamsCIs(fit) <--Somehow I got an s in 
between param & Cis.

I now get new error similarly as you, my apologies.

final <- MARSSparamCIs(fit)
Error in dpari[time.varying] <- dparmat(MLEobj, time.varying, t = t) :   
replacement has length zero

WHP

From: R-help  On Behalf Of Ashim Kapoor
Sent: Wednesday, January 23, 2019 1:38 AM
To: r-help@r-project.org
Subject: [R] Unable to compute Confidence Intervals from output from MARSS 
package

Dear All,

I am trying to use this package --->
https://cran.r-project.org/web/packages/MARSS/index.html
I am reading this book which shows some examples based on the above package
---> https://nwfsc-timeseries.github.io/atsa-labs/

In a few words, the incantation MARSS(...) estimates the parameters and
MARSSparamCIs should return the confidence intervals. My problem is that
MARSSparamCIs returns this error :-

> MARSSparamCIs(fit)
Error in MARSSharveyobsFI(MLEobj) : replacement has length zero
>

### This page and the next page, explains the jargon used :
https://nwfsc-timeseries.github.io/atsa-labs/sec-dlm-example-of-a-univariate-dlm.html

### Here is a MWE to recreate the error :-

# This is a Time Varying Parameters regression
# Here a is fixed and b is doing a RW.

x1 <- rnorm(1000)
b <- cumsum(rnorm(1000, sd = sqrt(.6)))
y <- 1 + b*x1 + rnorm(1000)

Z = array(NA,c(1,2,1000))
Z[1,1,] = rep(1,1000)
Z[1,2,] = x1

mod2.list = list ( B = matrix(c(1,0,0,1),nrow = 2 , byrow = T), U =
matrix(0,2,1),
Q = matrix(list(0,0,0,"s2b"),2,2), Z= Z, A = matrix(0), R = matrix("r"))

fit = MARSS(as.vector(y), model = mod2.list,inits =
list(x0=matrix(0,nrow=2,ncol=1)))

MARSSparamsCIs(fit)

# The above will give this error :
Error in MARSSharveyobsFI(MLEobj) : replacement has length zero

Best Regards,
Ashim

[[alternative HTML version deleted]]

__
mailto:R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Confidentiality Notice This message is sent from Zelis. ...{{dropped:13}}

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] Unable to compute Confidence Intervals from output from MARSS package

[Bill Poling]

sessionInfo()
#R version 3.5.2 (2018-12-20)
#Platform: x86_64-w64-mingw32/x64 (64-bit)
#Running under: Windows >= 8 x64 (build 9200)

Hello Ashim. I am not familiar with the MARSS pkg, however, I am always 
interested in following many of these R-Help questions and often run them for 
my own edification.

I ran your script and it appears to have run successfully, however, when I 
(naively) use the suggested "# Use MARSSparamCIs to compute CIs and bias 
estimates." below as you did I get a different error.

# I GET THIS ERROR
#Error in MARSSparamsCIs(fit) : could not find function "MARSSparamsCIs"

After further googling I find that I was (as suspected) incorrect in using " 
MARSSparamsCIs" as a function on fit or maybe not, take a look at this URL

https://rdrr.io/cran/MARSS/man/MARSSparamCIs.html

I'm not sure how to proceed but this is interesting to follow, thank you.

WHP

x1 <- rnorm(1000)
View(x1)#--Just curious
plot(x1)#--Just curious
b <- cumsum(rnorm(1000, sd = sqrt(.6)))
plot(b)#--Just curious
y <- 1 + b*x1 + rnorm(1000)
plot(y)#--Just curious

Z = array(NA,c(1,2,1000))
Z[1,1,] = rep(1,1000)
Z[1,2,] = x1

mod2.list = list ( B = matrix(c(1,0,0,1),nrow = 2 , byrow = T),
   U = matrix(0,2,1),
   Q = matrix(list(0,0,0,"s2b"),2,2),
   Z= Z, A = matrix(0), R = matrix("r"))
View(mod2.list)
#print(mod2.list)

str(mod2.list)#-KNOW THY DATA
#List of 6
# $ B: num [1:2, 1:2] 1 0 0 1
# $ U: num [1:2, 1] 0 0
# $ Q:List of 4
# ..$ : num 0
# ..$ : num 0
# ..$ : num 0
# ..$ : chr "s2b"
# ..- attr(*, "dim")= int [1:2] 2 2
# $ Z: num [1, 1:2, 1:1000] 1 1.2 1 1.04 1 ...
# $ A: num [1, 1] 0
# $ R: chr [1, 1] "r"

fit = MARSS(as.vector(y), model = mod2.list,inits = 
list(x0=matrix(0,nrow=2,ncol=1)))
#Success! abstol and log-log tests passed at 27 iterations.
# Alert: conv.test.slope.tol is 0.5.
# Test with smaller values (<0.1) to ensure convergence.
#
# MARSS fit is
# Estimation method: kem
# Convergence test: conv.test.slope.tol = 0.5, abstol = 0.001
# Estimation converged in 27 iterations.
# Log-likelihood: -1758.409
# AIC: 3524.818   AICc: 3524.858
#
# Estimate
# R.r  1.026
# Q.s2b0.606
# x0.X10.999
# x0.X2   -1.025
# Initial states (x0) defined at t=0
#
# Standard errors have not been calculated.
# Use MARSSparamCIs to compute CIs and bias estimates.

MARSSparamsCIs(fit)
#I GET THIS ERROR
#Error in MARSSparamsCIs(fit) : could not find function "MARSSparamsCIs"


#NOT THIS ERROR
# The above will give this error :
#Error in MARSSharveyobsFI(MLEobj) : replacement has length zero



WHP


From: R-help  On Behalf Of Ashim Kapoor
Sent: Wednesday, January 23, 2019 1:38 AM
To: r-help@r-project.org
Subject: [R] Unable to compute Confidence Intervals from output from MARSS 
package

Dear All,

I am trying to use this package --->
https://cran.r-project.org/web/packages/MARSS/index.html
I am reading this book which shows some examples based on the above package
---> https://nwfsc-timeseries.github.io/atsa-labs/

In a few words, the incantation MARSS(...) estimates the parameters and
MARSSparamCIs should return the confidence intervals. My problem is that
MARSSparamCIs returns this error :-

> MARSSparamCIs(fit)
Error in MARSSharveyobsFI(MLEobj) : replacement has length zero
>

### This page and the next page, explains the jargon used :
https://nwfsc-timeseries.github.io/atsa-labs/sec-dlm-example-of-a-univariate-dlm.html

### Here is a MWE to recreate the error :-

# This is a Time Varying Parameters regression
# Here a is fixed and b is doing a RW.

x1 <- rnorm(1000)
b <- cumsum(rnorm(1000, sd = sqrt(.6)))
y <- 1 + b*x1 + rnorm(1000)

Z = array(NA,c(1,2,1000))
Z[1,1,] = rep(1,1000)
Z[1,2,] = x1

mod2.list = list ( B = matrix(c(1,0,0,1),nrow = 2 , byrow = T), U =
matrix(0,2,1),
Q = matrix(list(0,0,0,"s2b"),2,2), Z= Z, A = matrix(0), R = matrix("r"))

fit = MARSS(as.vector(y), model = mod2.list,inits =
list(x0=matrix(0,nrow=2,ncol=1)))

MARSSparamsCIs(fit)

# The above will give this error :
Error in MARSSharveyobsFI(MLEobj) : replacement has length zero

Best Regards,
Ashim

[[alternative HTML version deleted]]

__
mailto:R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Confidentiality Notice This message is sent from Zelis. ...{{dropped:13}}

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] Packages

[Duncan Murdoch]

On 23/01/2019 12:27 p.m., AbouEl-Makarim Aboueissa wrote:

here is the messages I got when I install the "car" package:


You didn't install it, you got errors during the install.

I'm not sure why there was no attempt to install Rcpp (which was 
required by rio, see the error message).  Perhaps the mirror you used 
doesn't have it?  I'd recommend using the cloud.r-project.org mirror 
rather than a local one in almost any case.


In any case, Neal's advice to update R is likely to make your life a lot 
easier.


Duncan Murdoch



 > install.packages("car")
Installing package into ‘C:/Users/aaboueissa/Documents/R/win-library/3.3’
(as ‘lib’ is unspecified)
also installing the dependency ‘rio’


   There are binary versions available but the source versions are later:
     binary source needs_compilation
rio 0.5.10 0.5.16             FALSE
car  3.0-0  3.0-2             FALSE

installing the source packages ‘rio’, ‘car’

trying URL 'https://cran.case.edu/src/contrib/rio_0.5.16.tar.gz'
Content type 'application/x-gzip' length 420489 bytes (410 KB)
downloaded 410 KB

trying URL 'https://cran.case.edu/src/contrib/car_3.0-2.tar.gz'
Content type 'application/x-gzip' length 447952 bytes (437 KB)
downloaded 437 KB

* installing *source* package 'rio' ...
** package 'rio' successfully unpacked and MD5 sums checked
** R
** inst
** preparing package for lazy loading
Error in loadNamespace(j <- i[[1L]], c(lib.loc, .libPaths()), 
versionCheck = vI[[j]]) :

   there is no package called 'Rcpp'
ERROR: lazy loading failed for package 'rio'
* removing 'C:/Users/aaboueissa/Documents/R/win-library/3.3/rio'
ERROR: dependency 'rio' is not available for package 'car'
* removing 'C:/Users/aaboueissa/Documents/R/win-library/3.3/car'

The downloaded source packages are in
 
‘C:\Users\aaboueissa\AppData\Local\Temp\RtmpK0MQ8V\downloaded_packages’

Warning messages:
1: running command '"C:/PROGRA~1/R/R-33~1.2/bin/x64/R" CMD INSTALL -l 
"C:\Users\aaboueissa\Documents\R\win-library\3.3" 
C:\Users\AABOUE~1\AppData\Local\Temp\RtmpK0MQ8V/downloaded_packages/rio_0.5.16.tar.gz' 
had status 1

2: In install.packages("car") :
   installation of package ‘rio’ had non-zero exit status
3: running command '"C:/PROGRA~1/R/R-33~1.2/bin/x64/R" CMD INSTALL -l 
"C:\Users\aaboueissa\Documents\R\win-library\3.3" 
C:\Users\AABOUE~1\AppData\Local\Temp\RtmpK0MQ8V/downloaded_packages/car_3.0-2.tar.gz' 
had status 1

4: In install.packages("car") :
   installation of package ‘car’ had non-zero exit status


__

*AbouEl-Makarim Aboueissa, PhD
*
*
*
*Professor, Statistics and Data Science*
*Graduate Coordinator*
*Department of Mathematics and Statistics
*
*University of Southern Maine*



On Wed, Jan 23, 2019 at 12:20 PM Duncan Murdoch 
mailto:murdoch.dun...@gmail.com>> wrote:


On 23/01/2019 12:13 p.m., AbouEl-Makarim Aboueissa wrote:
 > Dear All:
 >
 > After installing the packages "car" and "alr3", I got the
following error
 > messages:
 >
 >
 >> library(car)
 > Error in library(car) : there is no package called ‘car’
 >
 >> library(alr3)
 > Error in library(alr3) : there is no package called ‘alr3’
 >
 > any helps would be appreciated.
 >

You need to show us the messages you received when you installed them.
The usual cause of problems like this is that you don't have write
permission on the default location, and R has chosen an alternate; then
when you try to attach the packages, you haven't told R to look in the
alternate location.

Duncan Murdoch



__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] Vectorizing a for-loop for cross-validation in R

[Berry, Charles]

See inline.

> On Jan 23, 2019, at 2:17 AM, Aleksandre Gavashelishvili 
>  wrote:
> 
> I'm trying to speed up a script that otherwise takes days to handle larger
> data sets. So, is there a way to completely vectorize or paralellize the
> following script:
> 
>*# k-fold cross validation*
> 
> df <- trees # a data frame 'trees' from R.
> df <- df[sample(nrow(df)), ] # randomly shuffles the data.
> k <- 10 # Number of folds. Note k=nrow(df) in the leave-one-out cross
> validation.
> folds <- cut(seq(from=1, to=nrow(df)), breaks=k, labels=FALSE) # creates
> unique numbers for k equally size folds.
> df$ID <- folds # adds fold IDs.
> df[paste("pred", 1:3, sep="")] <- NA # adds multiple columns "pred1"
> "pred2" "pred3" to speed up the following loop.
> 
> library(mgcv)
> 

Rprof()

replicate(100, {


> for(i in 1:k) {
>  # looping for different models:
>  m1 <- gam(Volume ~ s(Height), data=df, subset=(ID != i))
>  m2 <- gam(Volume ~ s(Girth), data=df, subset=(ID != i))
>  m3 <- gam(Volume ~ s(Girth) + s(Height), data=df, subset=(ID != i))
> 
>  # looping for predictions:
>  df[df$ID==i, "pred1"] <- predict(m1, df[df$ID==i, ], type="response")
>  df[df$ID==i, "pred2"] <- predict(m2, df[df$ID==i, ], type="response")
>  df[df$ID==i, "pred3"] <- predict(m3, df[df$ID==i, ], type="response")
> }
> 

})

Rprof(NULL)

summaryRprof()

## read ?Rprof to get a sense of what it does

## read the summary to determine where time is being spent.

## the result was surprising to me. YMMV.

## there may be redundancies that you can eliminate by 
##  - doing the setup within gam() one time and saving it
##  - calling the worker functions by modifying the setup 
##in a loop or function and saving the results


> # calculating residuals:
> df$res1 <- with(df, Volume - pred1)
> df$res2 <- with(df, Volume - pred2)
> df$res3 <- with(df, Volume - pred3)
> 
> Model <- paste("m", 1:3, sep="") # creates a vector of model names.
> 
> # creating a vector of mean-square errors (MSE):
> MSE <- with(df, c(
>  sum(res1^2) / nrow(df),
>  sum(res2^2) / nrow(df),
>  sum(res3^2) / nrow(df)
> ))
> 
> model.mse <- data.frame(Model, MSE) # creates a data frame of model names
> and mean-square errors.
> model.mse <- model.mse[order(model.mse$MSE), ] # rearranges the previous
> data frame in order of increasing mean-square errors.
> 
> I'd appreciate any help. This code takes several days if run on >=30,000
> different GAM models and 3 predictors. Could you please help with
> re-writing the script into sapply() or foreach()/doParallel format?
> 

This is something you should learn to do. It is pretty standard practice. Use 
the body of your for loop as the body of a function, add arguments, and create 
a suitable return value. The something like

lapply( 1:k, your.loop.body.function, other.arg1, other.arg2, ...)

should work.  If it does, then parallel::mclapply(...) should also work.

HTH,

Chuck

 
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] Packages

[Neal Fultz]

I'd recommend you upgrade to R version 3.5.2, the version you have is quite
out of date.

On Wed, Jan 23, 2019 at 9:42 AM AbouEl-Makarim Aboueissa <
abouelmakarim1...@gmail.com> wrote:

> here is the messages I got when I install the "car" package:
>
> > install.packages("car")
> Installing package into ‘C:/Users/aaboueissa/Documents/R/win-library/3.3’
> (as ‘lib’ is unspecified)
> also installing the dependency ‘rio’
>
>
>   There are binary versions available but the source versions are later:
> binary source needs_compilation
> rio 0.5.10 0.5.16 FALSE
> car  3.0-0  3.0-2 FALSE
>
> installing the source packages ‘rio’, ‘car’
>
> trying URL 'https://cran.case.edu/src/contrib/rio_0.5.16.tar.gz'
> Content type 'application/x-gzip' length 420489 bytes (410 KB)
> downloaded 410 KB
>
> trying URL 'https://cran.case.edu/src/contrib/car_3.0-2.tar.gz'
> Content type 'application/x-gzip' length 447952 bytes (437 KB)
> downloaded 437 KB
>
> * installing *source* package 'rio' ...
> ** package 'rio' successfully unpacked and MD5 sums checked
> ** R
> ** inst
> ** preparing package for lazy loading
> Error in loadNamespace(j <- i[[1L]], c(lib.loc, .libPaths()), versionCheck
> = vI[[j]]) :
>   there is no package called 'Rcpp'
> ERROR: lazy loading failed for package 'rio'
> * removing 'C:/Users/aaboueissa/Documents/R/win-library/3.3/rio'
> ERROR: dependency 'rio' is not available for package 'car'
> * removing 'C:/Users/aaboueissa/Documents/R/win-library/3.3/car'
>
> The downloaded source packages are in
>
> ‘C:\Users\aaboueissa\AppData\Local\Temp\RtmpK0MQ8V\downloaded_packages’
> Warning messages:
> 1: running command '"C:/PROGRA~1/R/R-33~1.2/bin/x64/R" CMD INSTALL -l
> "C:\Users\aaboueissa\Documents\R\win-library\3.3"
>
> C:\Users\AABOUE~1\AppData\Local\Temp\RtmpK0MQ8V/downloaded_packages/rio_0.5.16.tar.gz'
> had status 1
> 2: In install.packages("car") :
>   installation of package ‘rio’ had non-zero exit status
> 3: running command '"C:/PROGRA~1/R/R-33~1.2/bin/x64/R" CMD INSTALL -l
> "C:\Users\aaboueissa\Documents\R\win-library\3.3"
>
> C:\Users\AABOUE~1\AppData\Local\Temp\RtmpK0MQ8V/downloaded_packages/car_3.0-2.tar.gz'
> had status 1
> 4: In install.packages("car") :
>   installation of package ‘car’ had non-zero exit status
>
>
> __
>
>
> *AbouEl-Makarim Aboueissa, PhD*
>
> *Professor, Statistics and Data Science*
> *Graduate Coordinator*
>
> *Department of Mathematics and Statistics*
> *University of Southern Maine*
>
>
>
> On Wed, Jan 23, 2019 at 12:20 PM Duncan Murdoch 
> wrote:
>
> > On 23/01/2019 12:13 p.m., AbouEl-Makarim Aboueissa wrote:
> > > Dear All:
> > >
> > > After installing the packages "car" and "alr3", I got the following
> error
> > > messages:
> > >
> > >
> > >> library(car)
> > > Error in library(car) : there is no package called ‘car’
> > >
> > >> library(alr3)
> > > Error in library(alr3) : there is no package called ‘alr3’
> > >
> > > any helps would be appreciated.
> > >
> >
> > You need to show us the messages you received when you installed them.
> > The usual cause of problems like this is that you don't have write
> > permission on the default location, and R has chosen an alternate; then
> > when you try to attach the packages, you haven't told R to look in the
> > alternate location.
> >
> > Duncan Murdoch
> >
>
> [[alternative HTML version deleted]]
>
> __
> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

[[alternative HTML version deleted]]

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

[R] read_xl question

[Thomas Subia via R-help]

Colleagues,

 I have a workbook which has 3 worksheets

I need to extract data from two specific cells from one ofthose worksheets.

 

I can use read_excel to do this for one file.

data<-read_excel("C:/Desktop/Excel_raw_data/0020-49785 8768.xls",

sheet="Flow Data",range=("b9:c10"))

 

How can I do this for all my Excel files in the directory?

 

I can get the list of Excel files using: files =list.files(pattern="*.xls")

But I’m not sure where to go from here.

Some guidance would be appreciated.

 

All the best

 Thomas Subia

Thomas Subia

[[alternative HTML version deleted]]

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] Packages

[AbouEl-Makarim Aboueissa]

here is the messages I got when I install the "car" package:

> install.packages("car")
Installing package into ‘C:/Users/aaboueissa/Documents/R/win-library/3.3’
(as ‘lib’ is unspecified)
also installing the dependency ‘rio’


  There are binary versions available but the source versions are later:
binary source needs_compilation
rio 0.5.10 0.5.16 FALSE
car  3.0-0  3.0-2 FALSE

installing the source packages ‘rio’, ‘car’

trying URL 'https://cran.case.edu/src/contrib/rio_0.5.16.tar.gz'
Content type 'application/x-gzip' length 420489 bytes (410 KB)
downloaded 410 KB

trying URL 'https://cran.case.edu/src/contrib/car_3.0-2.tar.gz'
Content type 'application/x-gzip' length 447952 bytes (437 KB)
downloaded 437 KB

* installing *source* package 'rio' ...
** package 'rio' successfully unpacked and MD5 sums checked
** R
** inst
** preparing package for lazy loading
Error in loadNamespace(j <- i[[1L]], c(lib.loc, .libPaths()), versionCheck
= vI[[j]]) :
  there is no package called 'Rcpp'
ERROR: lazy loading failed for package 'rio'
* removing 'C:/Users/aaboueissa/Documents/R/win-library/3.3/rio'
ERROR: dependency 'rio' is not available for package 'car'
* removing 'C:/Users/aaboueissa/Documents/R/win-library/3.3/car'

The downloaded source packages are in

‘C:\Users\aaboueissa\AppData\Local\Temp\RtmpK0MQ8V\downloaded_packages’
Warning messages:
1: running command '"C:/PROGRA~1/R/R-33~1.2/bin/x64/R" CMD INSTALL -l
"C:\Users\aaboueissa\Documents\R\win-library\3.3"
C:\Users\AABOUE~1\AppData\Local\Temp\RtmpK0MQ8V/downloaded_packages/rio_0.5.16.tar.gz'
had status 1
2: In install.packages("car") :
  installation of package ‘rio’ had non-zero exit status
3: running command '"C:/PROGRA~1/R/R-33~1.2/bin/x64/R" CMD INSTALL -l
"C:\Users\aaboueissa\Documents\R\win-library\3.3"
C:\Users\AABOUE~1\AppData\Local\Temp\RtmpK0MQ8V/downloaded_packages/car_3.0-2.tar.gz'
had status 1
4: In install.packages("car") :
  installation of package ‘car’ had non-zero exit status


__


*AbouEl-Makarim Aboueissa, PhD*

*Professor, Statistics and Data Science*
*Graduate Coordinator*

*Department of Mathematics and Statistics*
*University of Southern Maine*



On Wed, Jan 23, 2019 at 12:20 PM Duncan Murdoch 
wrote:

> On 23/01/2019 12:13 p.m., AbouEl-Makarim Aboueissa wrote:
> > Dear All:
> >
> > After installing the packages "car" and "alr3", I got the following error
> > messages:
> >
> >
> >> library(car)
> > Error in library(car) : there is no package called ‘car’
> >
> >> library(alr3)
> > Error in library(alr3) : there is no package called ‘alr3’
> >
> > any helps would be appreciated.
> >
>
> You need to show us the messages you received when you installed them.
> The usual cause of problems like this is that you don't have write
> permission on the default location, and R has chosen an alternate; then
> when you try to attach the packages, you haven't told R to look in the
> alternate location.
>
> Duncan Murdoch
>

[[alternative HTML version deleted]]

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] Function in default parameter value closing over variables defined later in the enclosing function

[Jan T Kim via R-help]

Hi Duncan,

On Wed, Jan 23, 2019 at 10:02:00AM -0500, Duncan Murdoch wrote:
> On 23/01/2019 5:27 a.m., Jan T Kim wrote:
> >Hi Ivan & All,
> >
> >R's scoping system basically goes to all environments along the call
> >stack when trying to resolve an unbound variable, see the language
> >definition [1], section 4.3.4, and perhaps also 2.1.5.
> 
> You are misinterpreting that section.  It's not the call stack that is
> searched, it's the chain of environments that starts with the evaluation
> frame of the current function.  Those are very different.

yes -- I meant the environment chain but mistakenly wrote "call stack",
sorry. Thanks for pointing this out.

Best regards, Jan


> For example,
> 
> 
> g <- function() {
>   print(secret)
> }
> 
> f <- function() {
>   secret <- "secret"
>   g()
> }
> 
> would fail, because even though secret is defined in the caller of g() and
> is therefore in the call stack, that's irrelevant:  it's not in g's
> evaluation frame (which has no variables) or its parent (which is the global
> environment if those definitions were evaluated there).
> 
> Duncan Murdoch
> 
> >
> >Generally, unbound variables should be used with care. It's a bit
> >difficult to decide whether and how the code should be rewritten,
> >I'd say that depends on the underlying intentions / purposes. As it
> >is, the code could be simplified to just
> >
> > print("secret");
> >
> >but that's probably missing the point.
> >
> >Best regards, Jan
> >
> >
> >[1] https://cran.r-project.org/doc/manuals/r-release/R-lang.html
> >
> >On Wed, Jan 23, 2019 at 12:53:01PM +0300, Ivan Krylov wrote:
> >>Hi!
> >>
> >>I needed to generalize a loss function being optimized inside another
> >>function, so I made it a function argument with a default value. It
> >>worked without problems, but later I noticed that the inner function,
> >>despite being defined in the function arguments, somehow closes over a
> >>variable belonging to the outer function, which is defined later.
> >>
> >>Example:
> >>
> >>outside <- function(inside = function() print(secret)) {
> >>secret <- 'secret'
> >>inside()
> >>}
> >>outside()
> >>
> >>I'm used to languages that have both lambdas and variable declaration
> >>(like perl5 -Mstrict or C++11), so I was a bit surprised.
> >>
> >>Does this work because R looks up the variable by name late enough at
> >>runtime for the `secret` variable to exist in the parent environment of
> >>the `inside` function? Can I rely on it? Is this considered bad style?
> >>Should I rewrite it (and how)?
> >>
> >>-- 
> >>Best regards,
> >>Ivan
> >>
> >>__
> >>R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> >>https://stat.ethz.ch/mailman/listinfo/r-help
> >>PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> >>and provide commented, minimal, self-contained, reproducible code.
> >
> >__
> >R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> >https://stat.ethz.ch/mailman/listinfo/r-help
> >PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> >and provide commented, minimal, self-contained, reproducible code.
> >
> 
> __
> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] Packages

[Duncan Murdoch]

On 23/01/2019 12:13 p.m., AbouEl-Makarim Aboueissa wrote:

Dear All:

After installing the packages "car" and "alr3", I got the following error
messages:



library(car)

Error in library(car) : there is no package called ‘car’


library(alr3)

Error in library(alr3) : there is no package called ‘alr3’

any helps would be appreciated.



You need to show us the messages you received when you installed them. 
The usual cause of problems like this is that you don't have write 
permission on the default location, and R has chosen an alternate; then 
when you try to attach the packages, you haven't told R to look in the 
alternate location.


Duncan Murdoch

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

[R] Packages

[AbouEl-Makarim Aboueissa]

Dear All:

After installing the packages "car" and "alr3", I got the following error
messages:


> library(car)
Error in library(car) : there is no package called ‘car’

> library(alr3)
Error in library(alr3) : there is no package called ‘alr3’

any helps would be appreciated.


with many thanks
abou
__


*AbouEl-Makarim Aboueissa, PhD*

*Professor, Statistics and Data Science*
*Graduate Coordinator*

*Department of Mathematics and Statistics*
*University of Southern Maine*

[[alternative HTML version deleted]]

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

[R] Vectorizing a for-loop for cross-validation in R

[Aleksandre Gavashelishvili]

I'm trying to speed up a script that otherwise takes days to handle larger
data sets. So, is there a way to completely vectorize or paralellize the
following script:

*# k-fold cross validation*

df <- trees # a data frame 'trees' from R.
df <- df[sample(nrow(df)), ] # randomly shuffles the data.
k <- 10 # Number of folds. Note k=nrow(df) in the leave-one-out cross
validation.
folds <- cut(seq(from=1, to=nrow(df)), breaks=k, labels=FALSE) # creates
unique numbers for k equally size folds.
df$ID <- folds # adds fold IDs.
df[paste("pred", 1:3, sep="")] <- NA # adds multiple columns "pred1"
"pred2" "pred3" to speed up the following loop.

library(mgcv)

for(i in 1:k) {
  # looping for different models:
  m1 <- gam(Volume ~ s(Height), data=df, subset=(ID != i))
  m2 <- gam(Volume ~ s(Girth), data=df, subset=(ID != i))
  m3 <- gam(Volume ~ s(Girth) + s(Height), data=df, subset=(ID != i))

  # looping for predictions:
  df[df$ID==i, "pred1"] <- predict(m1, df[df$ID==i, ], type="response")
  df[df$ID==i, "pred2"] <- predict(m2, df[df$ID==i, ], type="response")
  df[df$ID==i, "pred3"] <- predict(m3, df[df$ID==i, ], type="response")
}

# calculating residuals:
df$res1 <- with(df, Volume - pred1)
df$res2 <- with(df, Volume - pred2)
df$res3 <- with(df, Volume - pred3)

Model <- paste("m", 1:3, sep="") # creates a vector of model names.

# creating a vector of mean-square errors (MSE):
MSE <- with(df, c(
  sum(res1^2) / nrow(df),
  sum(res2^2) / nrow(df),
  sum(res3^2) / nrow(df)
))

model.mse <- data.frame(Model, MSE) # creates a data frame of model names
and mean-square errors.
model.mse <- model.mse[order(model.mse$MSE), ] # rearranges the previous
data frame in order of increasing mean-square errors.

I'd appreciate any help. This code takes several days if run on >=30,000
different GAM models and 3 predictors. Could you please help with
re-writing the script into sapply() or foreach()/doParallel format?

Thanks
Lexo

[[alternative HTML version deleted]]

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] calculating quintile values of numeric data?

[David L Carlson]

Also quantile() and cut(). The only tricky part is making sure the minimum and 
maximum values are included.

> set.seed(42)
> x <- rnorm(100, 25, 3)
> bks <- quantile(x, prob=c(0, .2, .4, .6, .8, 1))
> y <- cut(x, breaks=bks, labels=1:5, include.lowest=TRUE)
> table(y)
y
 1  2  3  4  5 
20 20 20 20 20
> z <- findInterval(x, bks, all.inside=TRUE)
> table(z)
z
 1  2  3  4  5 
20 20 20 20 20


David L Carlson
Department of Anthropology
Texas A&M University
College Station, TX 77843-4352


-Original Message-
From: R-help  On Behalf Of Jeff Newmiller
Sent: Wednesday, January 23, 2019 1:00 AM
To: r-help@r-project.org; Kelly Thompson 
Subject: Re: [R] calculating quintile values of numeric data?

?range
?findInterval

On January 22, 2019 6:54:14 PM PST, Kelly Thompson  wrote:
>I’d like to take numeric data, and calculate numeric “quintiles” with
>integer values in from 1 – 5 , with values in the lowest 20% of values
>having a value of 1, the >20 - <= 40% of values having a value of 2,
>the >40% - <=60% of values having a value of 3, etc.
>
>How can I use quantcut, or another function, to do this?
>
>
>Thanks!
>
>
>Ex.
>
>x <- c(1:10)
>
>I want:
>myquintilefunction (x, q=5, na.rm=T) to return a vector with values:
>1,1,2,2,3,3,4,4,5,5
>
>Thanks!
>
>__
>R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
>https://stat.ethz.ch/mailman/listinfo/r-help
>PLEASE do read the posting guide
>http://www.R-project.org/posting-guide.html
>and provide commented, minimal, self-contained, reproducible code.

-- 
Sent from my phone. Please excuse my brevity.

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] Function in default parameter value closing over variables defined later in the enclosing function

[Duncan Murdoch]

On 23/01/2019 5:27 a.m., Jan T Kim wrote:

Hi Ivan & All,

R's scoping system basically goes to all environments along the call
stack when trying to resolve an unbound variable, see the language
definition [1], section 4.3.4, and perhaps also 2.1.5.


You are misinterpreting that section.  It's not the call stack that is 
searched, it's the chain of environments that starts with the evaluation 
frame of the current function.  Those are very different.  For example,



g <- function() {
  print(secret)
}

f <- function() {
  secret <- "secret"
  g()
}

would fail, because even though secret is defined in the caller of g() 
and is therefore in the call stack, that's irrelevant:  it's not in g's 
evaluation frame (which has no variables) or its parent (which is the 
global environment if those definitions were evaluated there).


Duncan Murdoch



Generally, unbound variables should be used with care. It's a bit
difficult to decide whether and how the code should be rewritten,
I'd say that depends on the underlying intentions / purposes. As it
is, the code could be simplified to just

 print("secret");

but that's probably missing the point.

Best regards, Jan


[1] https://cran.r-project.org/doc/manuals/r-release/R-lang.html

On Wed, Jan 23, 2019 at 12:53:01PM +0300, Ivan Krylov wrote:

Hi!

I needed to generalize a loss function being optimized inside another
function, so I made it a function argument with a default value. It
worked without problems, but later I noticed that the inner function,
despite being defined in the function arguments, somehow closes over a
variable belonging to the outer function, which is defined later.

Example:

outside <- function(inside = function() print(secret)) {
secret <- 'secret'
inside()
}
outside()

I'm used to languages that have both lambdas and variable declaration
(like perl5 -Mstrict or C++11), so I was a bit surprised.

Does this work because R looks up the variable by name late enough at
runtime for the `secret` variable to exist in the parent environment of
the `inside` function? Can I rely on it? Is this considered bad style?
Should I rewrite it (and how)?

--
Best regards,
Ivan

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.


__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.



__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] Function in default parameter value closing over variables defined later in the enclosing function

[Duncan Murdoch]

On 23/01/2019 4:53 a.m., Ivan Krylov wrote:

Hi!

I needed to generalize a loss function being optimized inside another
function, so I made it a function argument with a default value. It
worked without problems, but later I noticed that the inner function,
despite being defined in the function arguments, somehow closes over a
variable belonging to the outer function, which is defined later.

Example:

outside <- function(inside = function() print(secret)) {
secret <- 'secret'
inside()
}
outside()

I'm used to languages that have both lambdas and variable declaration
(like perl5 -Mstrict or C++11), so I was a bit surprised.


Defaults of variables are evaluated in the evaluation frame of the call.
So the inside() function is created in the evaluation frame, and it's 
environment will be that frame.


When it is called it will create a new evaluation frame (empty in your 
example), with a parent being its environment, i.e. the evaluation frame 
from when it was created, so it will be able to see your secret variable.


If it made an assignment to secret using standard "<-" assignment, it 
would create a new variable in its own evaluation frame, but if it used 
superassignment "<<-", it would modify the original secret variable.




Does this work because R looks up the variable by name late enough at
runtime for the `secret` variable to exist in the parent environment of
the `inside` function? Can I rely on it? Is this considered bad style?
Should I rewrite it (and how)?


I would consider it bad style if the inside() function had anything 
other than a trivial definition as in your example.  However, in my 
opinion it would be fine to write it as


 outside <- function(inside = defaultInsideFn) {
defaultInsideFn <- function() print(secret)
secret <- 'secret'
inside()
 }

which is essentially equivalent, other than having a shorter header on 
the outside() function.


Duncan Murdoch

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] large number of scrollable histograms....

[Bill Poling]

Hi akshay Kulkarni, I just worked through this great tutorial the other day, 
hope this helps!

WHP

https://www.r-bloggers.com/how-to-combine-multiple-ggplot-plots-to-make-publication-ready-plots/



From: R-help  On Behalf Of Eric Berger
Sent: Tuesday, January 22, 2019 8:59 AM
To: PIKAL Petr 
Cc: R help Mailing list 
Subject: Re: [R] large number of scrollable histograms

Another alternative is to use ggplot2 to create the various plots, then put
them into
a list and use cowplot::plot_grid to plot the grid of plots with a
specified number of
rows and columns. Here's some pseudo code to give you the general idea

Step 1: generate the plots and put them into a list
for ( i in 1:(nrow*ncol) )
pL[[ i ]] <- ggplot( ... )

Step 2: display the plots in a grid
print( cowplot::plot_grid( plotlist=pL, nrow=nrow, ncol=ncol ) )

On Tue, Jan 22, 2019 at 3:49 PM PIKAL Petr  
wrote:

> Hi
>
> what about to create all histograms in pdf device?
>
> see
> ?pdf
>
> Cheers
> Petr
>
>
> > -Original Message-
> > From: R-help  On Behalf Of akshay 
> > kulkarni
> > Sent: Tuesday, January 22, 2019 2:21 PM
> > To: R help Mailing list 
> > Subject: [R] large number of scrollable histograms
> >
> > dear members,
> > I am a day trader based in INDIA. I use R
> for my research.
> >
> > I have about 200 vectors whose histograms I need to inspect. I have to
> > compare them simultaneously.
> >
> > I know methods whereby you can plot multiple histograms on one screen.
> > However, you can clearly view only 4 to 5 histograms in one screen.
> >
> > Is there a way to construct a long list of all the 100 histograms that
> can be
> > scrollable (like you scroll up or down the R console) both downwards and
> > upwards? Any package to that effect?
> >
> > I would be highly grateful, also, if you can offer any suggestions or
> "out of the
> > box" ideas to simultaneously compare all the 100 histograms.
> >
> > very many thanks for your help and support..
> > yours sincerely,
> > AKSHAY M KULKARNI
> >
> > [[alternative HTML version deleted]]
> >
> > __
> > mailto:R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide http://www.R-project.org/posting-
> > guide.html
> > and provide commented, minimal, self-contained, reproducible code.
> Osobní údaje: Informace o zpracování a ochraně osobních údajů obchodních
> partnerů PRECHEZA a.s. jsou zveřejněny na:
> https://www.precheza.cz/zasady-ochrany-osobnich-udaju/ | Information
> about processing and protection of business partner’s personal data are
> available on website:
> https://www.precheza.cz/en/personal-data-protection-principles/
> Důvěrnost: Tento e-mail a jakékoliv k němu připojené dokumenty jsou
> důvěrné a podléhají tomuto právně závaznému prohláąení o vyloučení
> odpovědnosti: https://www.precheza.cz/01-dovetek/ | This email and any
> documents attached to it may be confidential and are subject to the legally
> binding disclaimer: https://www.precheza.cz/en/01-disclaimer/
>
> __
> mailto:R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

[[alternative HTML version deleted]]

__
mailto:R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Confidentiality Notice This message is sent from Zelis. This transmission may 
contain information which is privileged and confidential and is intended for 
the personal and confidential use of the named recipient only. Such information 
may be protected by applicable State and Federal laws from this disclosure or 
unauthorized use. If the reader of this message is not the intended recipient, 
or the employee or agent responsible for delivering the message to the intended 
recipient, you are hereby notified that any disclosure, review, discussion, 
copying, or taking any action in reliance on the contents of this transmission 
is strictly prohibited. If you have received this transmission in error, please 
contact the sender immediately. Zelis, 2018.
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] Function in default parameter value closing over variables defined later in the enclosing function

[Jan T Kim]

Hi Ivan & All,

R's scoping system basically goes to all environments along the call
stack when trying to resolve an unbound variable, see the language
definition [1], section 4.3.4, and perhaps also 2.1.5.

Generally, unbound variables should be used with care. It's a bit
difficult to decide whether and how the code should be rewritten,
I'd say that depends on the underlying intentions / purposes. As it
is, the code could be simplified to just

print("secret");

but that's probably missing the point.

Best regards, Jan


[1] https://cran.r-project.org/doc/manuals/r-release/R-lang.html

On Wed, Jan 23, 2019 at 12:53:01PM +0300, Ivan Krylov wrote:
> Hi!
> 
> I needed to generalize a loss function being optimized inside another
> function, so I made it a function argument with a default value. It
> worked without problems, but later I noticed that the inner function,
> despite being defined in the function arguments, somehow closes over a
> variable belonging to the outer function, which is defined later.
> 
> Example:
> 
> outside <- function(inside = function() print(secret)) {
>   secret <- 'secret'
>   inside()
> }
> outside()
> 
> I'm used to languages that have both lambdas and variable declaration
> (like perl5 -Mstrict or C++11), so I was a bit surprised.
> 
> Does this work because R looks up the variable by name late enough at
> runtime for the `secret` variable to exist in the parent environment of
> the `inside` function? Can I rely on it? Is this considered bad style? 
> Should I rewrite it (and how)?
> 
> -- 
> Best regards,
> Ivan
> 
> __
> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

[R] Function in default parameter value closing over variables defined later in the enclosing function

[Ivan Krylov]

Hi!

I needed to generalize a loss function being optimized inside another
function, so I made it a function argument with a default value. It
worked without problems, but later I noticed that the inner function,
despite being defined in the function arguments, somehow closes over a
variable belonging to the outer function, which is defined later.

Example:

outside <- function(inside = function() print(secret)) {
secret <- 'secret'
inside()
}
outside()

I'm used to languages that have both lambdas and variable declaration
(like perl5 -Mstrict or C++11), so I was a bit surprised.

Does this work because R looks up the variable by name late enough at
runtime for the `secret` variable to exist in the parent environment of
the `inside` function? Can I rely on it? Is this considered bad style? 
Should I rewrite it (and how)?

-- 
Best regards,
Ivan

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] large number of scrollable histograms....

[PIKAL Petr]

Hi

Yes, you should get multipage pdf, each page populated by single call to plot 
function.

However I am not sur if your proposal with function will work.

I usually do simply



pdf("sample.pdf", 7, 5)

for (i in 1:n) {

hist(L[[i]])

}

dev.off()

Cheers
Petr

From: akshay kulkarni 
Sent: Wednesday, January 23, 2019 8:01 AM
To: PIKAL Petr ; R help Mailing list 

Subject: Re: large number of scrollable histograms

dear pikal,
Thanks for the suggestion
I have checked ?pdf.  The syntax is:


> pdf("sample.pdf", 7, 5)

> hist(vector1)

> dev.off()
So, I surmise that instead of one hist function in the second line, if I 
include multiple histograms through a loop, it would write ALL histograms to 
sample.pdf (something like this):


> pdf("sample.pdf", 7, 5)

> histlist

> dev.off()

> histlist <- function(){ for( i in 1:length(L)){histlist[[i]] <- hist(L[[i]]}; 
> return(histlist)}



Am I right? Would this pdf document(sample.pdf)  be scrollable?



very many thanks for your time
yours sincerely,
AKSHAY M KULKARNI


From: PIKAL Petr mailto:petr.pi...@precheza.cz>>
Sent: Tuesday, January 22, 2019 7:19 PM
To: akshay kulkarni; R help Mailing list
Subject: RE: large number of scrollable histograms

Hi

what about to create all histograms in pdf device?

see
?pdf

Cheers
Petr


> -Original Message-
> From: R-help 
> mailto:r-help-boun...@r-project.org>> On Behalf 
> Of akshay kulkarni
> Sent: Tuesday, January 22, 2019 2:21 PM
> To: R help Mailing list mailto:r-help@r-project.org>>
> Subject: [R] large number of scrollable histograms
>
> dear members,
> I am a day trader based in INDIA. I use R for my 
> research.
>
> I have about 200 vectors whose histograms I need to inspect. I have to
> compare them simultaneously.
>
> I know methods whereby you can plot multiple histograms on one screen.
> However, you can clearly view only 4 to 5  histograms in one screen.
>
> Is there a way to construct a long list of all the 100 histograms that can be
> scrollable (like you scroll up or down the R console) both downwards and
> upwards? Any package to that effect?
>
> I would be highly grateful, also, if you can offer any suggestions or "out of 
> the
> box" ideas to simultaneously compare all the 100 histograms.
>
> very many thanks for your help and support..
> yours sincerely,
> AKSHAY M KULKARNI
>
> [[alternative HTML version deleted]]
>
> __
> R-help@r-project.org mailing list -- To 
> UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-
> guide.html
> and provide commented, minimal, self-contained, reproducible code.
Osobn� �daje: Informace o zpracov�n� a ochran� osobn�ch �daj� obchodn�ch 
partner� PRECHEZA a.s. jsou zve�ejn�ny na: 
https://www.precheza.cz/zasady-ochrany-osobnich-udaju/ | Information about 
processing and protection of business partner's personal data are available on 
website: https://www.precheza.cz/en/personal-data-protection-principles/
D�v�rnost: Tento e-mail a jak�koliv k n�mu p�ipojen� dokumenty jsou d�v�rn� a 
podl�haj� tomuto pr�vn� z�vazn�mu prohl�en� o vylou�en� odpov�dnosti: 
https://www.precheza.cz/01-dovetek/ | This email and any documents attached to 
it may be confidential and are subject to the legally binding disclaimer: 
https://www.precheza.cz/en/01-disclaimer/

[[alternative HTML version deleted]]

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] calculating quintile values of numeric data? Alexandros Kouretsis

[Alexandros Kouretsis]

cut can do the job

q_prob <- seq(0, 1, 0.2)
cut(x, breaks = quantile(x, probs = q_prob), include.lowest = T , labels =
1:5)

[[alternative HTML version deleted]]

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.