Re: [R] R getting "Killed" while running VAR model

2016-06-08 Thread Jeff Newmiller 
1. Don't allocate it. 

2. If it was, would it make a difference? 

Seriously, some algorithms need more memory than others, and some packages are 
more wasteful than others. R is not monolithic... sometimes you just have to 
roll up your sleeves or buy more memory. 
-- 
Sent from my phone. Please excuse my brevity.

On June 8, 2016 7:51:00 AM PDT, Vivek Singh  wrote:
>I checked the issue on different forums like stackoverflow. The issue
>is
>related to Out Of Memory (OOM) linux feature which kills processes that
>consume large memory and swap. I started to monitor the memory and swap
>consumption while VAR model was running. The R consumed all 32 GB of
>RAM
>memory and then 20 GB of swap. I have only 32GB of RAM and 20GB of swap
>in
>my server. Then the R was killed.
>
>So, i have two queries:
>
>1. Is there a way to limit the amount of memory used by R so that it
>runs
>for longer time?
>
>2. Is it some sort of inefficiency of garbage collector in R?
>
>
>
>Regards,
>
>Vivek Kumar Singh
>
>PhD student,
>Information Systems Decision Sciences,
>MUMA College of Business,
>USF
>Phone- (813) 5809131
>Web: http://vivek4.myweb.usf.edu/
>
>On Wed, Jun 8, 2016 at 10:13 AM, Vivek Singh 
>wrote:
>
>> I am using *R version 3.0.2* (2013-09-25) on Ubuntu desktop (*Ubuntu
>> 14.04.4 LTS*). I am running *var model *on a matrix with 199 columns
>and
>> 604800 rows. The server has 12 core and 32GB of memory. When the
>model is
>> running, i checked CPU and Memory consumption using 'htop' linux
>command. I
>> observe all the cores are being used and memory memory usage is on
>average
>> 17GB out of 32GB. After running the model for about an hour, the
>system
>> kills the R process. Following the output:
>>
>> *> library("vars")*
>> *> vmodel=VAR(only_variables_without_missing, p = 1, type ="both")*
>> *Killed*
>>
>> Please help.
>>
>>
>>
>> Regards,
>>
>> Vivek Kumar Singh
>>
>> PhD student,
>> Information Systems Decision Sciences,
>> MUMA College of Business,
>> USF
>> Phone- (813) 5809131
>> Web: http://vivek4.myweb.usf.edu/
>>
>> On Tue, May 31, 2016 at 4:41 PM, Uwe Ligges <
>> lig...@statistik.tu-dortmund.de> wrote:
>>
>>> Wild guess: You have huge and high dimensional VAR models, i.e. the
>>> matrices get huge and you use huge amounts of memory and you use
>more than
>>> what is available physically. The operating system protects itself
>by
>>> killing processes in such a case...
>>>
>>> Best,
>>> Uwe Ligges
>>>
>>>
>>>
>>> On 31.05.2016 20:29, Vivek Singh wrote:
>>>
 Hi,

 I am using VARS (vector autoregressive model). The process gets
>killed
 after running for sometime. Following is the output of R.

 vivek@isds-research:~/cloudAuction/padding/panel$ cat var.Rout

 R version 3.0.2 (2013-09-25) -- "Frisbee Sailing"
 Copyright (C) 2013 The R Foundation for Statistical Computing
 Platform: x86_64-pc-linux-gnu (64-bit)

 R is free software and comes with ABSOLUTELY NO WARRANTY.
 You are welcome to redistribute it under certain conditions.
 Type 'license()' or 'licence()' for distribution details.

   Natural language support but running in an English locale

 R is a collaborative project with many contributors.
 Type 'contributors()' for more information and
 'citation()' on how to cite R or R packages in publications.

 Type 'demo()' for some demos, 'help()' for on-line help, or
 'help.start()' for an HTML browser interface to help.
 Type 'q()' to quit R.

 [Previously saved workspace restored]

 data=read.csv("output1.csv")
> attach(data)
> only_variables= subset(data, select=c(-date,-hour,-minute,-sec))
>
> library("vars")
>
 Loading required package: MASS
 Loading required package: strucchange
 Loading required package: zoo

 Attaching package: ‘zoo’

 The following objects are masked from ‘package:base’:

 as.Date, as.Date.numeric

 Loading required package: sandwich
 Loading required package: urca
 Loading required package: lmtest

> summary(VAR(only_variables, p = 1, type ="both"))
>
 *Killed*

 [[alternative HTML version deleted]]

 __
 R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
 https://stat.ethz.ch/mailman/listinfo/r-help
 PLEASE do read the posting guide
 http://www.R-project.org/posting-guide.html
 and provide commented, minimal, self-contained, reproducible code.


>>
>
>   [[alternative HTML version deleted]]
>
>__
>R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
>https://stat.ethz.ch/mailman/listinfo/r-help
>PLEASE do read the posting guide
>http://www.R-project.org/posting-guide.html
>and provide commented, minimal, self-contained, reproducible code.

[[alternative HTML

Re: [R] R getting "Killed" while running VAR model

2016-06-08 Thread Vivek Singh 
I checked the issue on different forums like stackoverflow. The issue is
related to Out Of Memory (OOM) linux feature which kills processes that
consume large memory and swap. I started to monitor the memory and swap
consumption while VAR model was running. The R consumed all 32 GB of RAM
memory and then 20 GB of swap. I have only 32GB of RAM and 20GB of swap in
my server. Then the R was killed.

So, i have two queries:

1. Is there a way to limit the amount of memory used by R so that it runs
for longer time?

2. Is it some sort of inefficiency of garbage collector in R?



Regards,

Vivek Kumar Singh

PhD student,
Information Systems Decision Sciences,
MUMA College of Business,
USF
Phone- (813) 5809131
Web: http://vivek4.myweb.usf.edu/

On Wed, Jun 8, 2016 at 10:13 AM, Vivek Singh  wrote:

> I am using *R version 3.0.2* (2013-09-25) on Ubuntu desktop (*Ubuntu
> 14.04.4 LTS*). I am running *var model *on a matrix with 199 columns and
> 604800 rows. The server has 12 core and 32GB of memory. When the model is
> running, i checked CPU and Memory consumption using 'htop' linux command. I
> observe all the cores are being used and memory memory usage is on average
> 17GB out of 32GB. After running the model for about an hour, the system
> kills the R process. Following the output:
>
> *> library("vars")*
> *> vmodel=VAR(only_variables_without_missing, p = 1, type ="both")*
> *Killed*
>
> Please help.
>
>
>
> Regards,
>
> Vivek Kumar Singh
>
> PhD student,
> Information Systems Decision Sciences,
> MUMA College of Business,
> USF
> Phone- (813) 5809131
> Web: http://vivek4.myweb.usf.edu/
>
> On Tue, May 31, 2016 at 4:41 PM, Uwe Ligges <
> lig...@statistik.tu-dortmund.de> wrote:
>
>> Wild guess: You have huge and high dimensional VAR models, i.e. the
>> matrices get huge and you use huge amounts of memory and you use more than
>> what is available physically. The operating system protects itself by
>> killing processes in such a case...
>>
>> Best,
>> Uwe Ligges
>>
>>
>>
>> On 31.05.2016 20:29, Vivek Singh wrote:
>>
>>> Hi,
>>>
>>> I am using VARS (vector autoregressive model). The process gets killed
>>> after running for sometime. Following is the output of R.
>>>
>>> vivek@isds-research:~/cloudAuction/padding/panel$ cat var.Rout
>>>
>>> R version 3.0.2 (2013-09-25) -- "Frisbee Sailing"
>>> Copyright (C) 2013 The R Foundation for Statistical Computing
>>> Platform: x86_64-pc-linux-gnu (64-bit)
>>>
>>> R is free software and comes with ABSOLUTELY NO WARRANTY.
>>> You are welcome to redistribute it under certain conditions.
>>> Type 'license()' or 'licence()' for distribution details.
>>>
>>>   Natural language support but running in an English locale
>>>
>>> R is a collaborative project with many contributors.
>>> Type 'contributors()' for more information and
>>> 'citation()' on how to cite R or R packages in publications.
>>>
>>> Type 'demo()' for some demos, 'help()' for on-line help, or
>>> 'help.start()' for an HTML browser interface to help.
>>> Type 'q()' to quit R.
>>>
>>> [Previously saved workspace restored]
>>>
>>> data=read.csv("output1.csv")
 attach(data)
 only_variables= subset(data, select=c(-date,-hour,-minute,-sec))

 library("vars")

>>> Loading required package: MASS
>>> Loading required package: strucchange
>>> Loading required package: zoo
>>>
>>> Attaching package: ‘zoo’
>>>
>>> The following objects are masked from ‘package:base’:
>>>
>>> as.Date, as.Date.numeric
>>>
>>> Loading required package: sandwich
>>> Loading required package: urca
>>> Loading required package: lmtest
>>>
 summary(VAR(only_variables, p = 1, type ="both"))

>>> *Killed*
>>>
>>> [[alternative HTML version deleted]]
>>>
>>> __
>>> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>> PLEASE do read the posting guide
>>> http://www.R-project.org/posting-guide.html
>>> and provide commented, minimal, self-contained, reproducible code.
>>>
>>>
>

[[alternative HTML version deleted]]

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] R getting "Killed" while running VAR model

2016-06-08 Thread Vivek Singh 
I am using *R version 3.0.2* (2013-09-25) on Ubuntu desktop (*Ubuntu
14.04.4 LTS*). I am running *var model *on a matrix with 199 columns and
604800 rows. The server has 12 core and 32GB of memory. When the model is
running, i checked CPU and Memory consumption using 'htop' linux command. I
observe all the cores are being used and memory memory usage is on average
17GB out of 32GB. After running the model for about an hour, the system
kills the R process. Following the output:

*> library("vars")*
*> vmodel=VAR(only_variables_without_missing, p = 1, type ="both")*
*Killed*

Please help.



Regards,

Vivek Kumar Singh

PhD student,
Information Systems Decision Sciences,
MUMA College of Business,
USF
Phone- (813) 5809131
Web: http://vivek4.myweb.usf.edu/

On Tue, May 31, 2016 at 4:41 PM, Uwe Ligges  wrote:

> Wild guess: You have huge and high dimensional VAR models, i.e. the
> matrices get huge and you use huge amounts of memory and you use more than
> what is available physically. The operating system protects itself by
> killing processes in such a case...
>
> Best,
> Uwe Ligges
>
>
>
> On 31.05.2016 20:29, Vivek Singh wrote:
>
>> Hi,
>>
>> I am using VARS (vector autoregressive model). The process gets killed
>> after running for sometime. Following is the output of R.
>>
>> vivek@isds-research:~/cloudAuction/padding/panel$ cat var.Rout
>>
>> R version 3.0.2 (2013-09-25) -- "Frisbee Sailing"
>> Copyright (C) 2013 The R Foundation for Statistical Computing
>> Platform: x86_64-pc-linux-gnu (64-bit)
>>
>> R is free software and comes with ABSOLUTELY NO WARRANTY.
>> You are welcome to redistribute it under certain conditions.
>> Type 'license()' or 'licence()' for distribution details.
>>
>>   Natural language support but running in an English locale
>>
>> R is a collaborative project with many contributors.
>> Type 'contributors()' for more information and
>> 'citation()' on how to cite R or R packages in publications.
>>
>> Type 'demo()' for some demos, 'help()' for on-line help, or
>> 'help.start()' for an HTML browser interface to help.
>> Type 'q()' to quit R.
>>
>> [Previously saved workspace restored]
>>
>> data=read.csv("output1.csv")
>>> attach(data)
>>> only_variables= subset(data, select=c(-date,-hour,-minute,-sec))
>>>
>>> library("vars")
>>>
>> Loading required package: MASS
>> Loading required package: strucchange
>> Loading required package: zoo
>>
>> Attaching package: ‘zoo’
>>
>> The following objects are masked from ‘package:base’:
>>
>> as.Date, as.Date.numeric
>>
>> Loading required package: sandwich
>> Loading required package: urca
>> Loading required package: lmtest
>>
>>> summary(VAR(only_variables, p = 1, type ="both"))
>>>
>> *Killed*
>>
>> [[alternative HTML version deleted]]
>>
>> __
>> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>>
>>

[[alternative HTML version deleted]]

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] R getting "Killed" while running VAR model

2016-05-31 Thread Bert Gunter 
Standard reply (see posting guide):

Update to the current version of R (3.3.0 or so) and retry. Your
version is old -- this often leads to incompatibilities with newer
software versions.

Cheers,
Bert

Bert Gunter

"The trouble with having an open mind is that people keep coming along
and sticking things into it."
-- Opus (aka Berkeley Breathed in his "Bloom County" comic strip )


On Tue, May 31, 2016 at 11:29 AM, Vivek Singh  wrote:
> Hi,
>
> I am using VARS (vector autoregressive model). The process gets killed
> after running for sometime. Following is the output of R.
>
> vivek@isds-research:~/cloudAuction/padding/panel$ cat var.Rout
>
> R version 3.0.2 (2013-09-25) -- "Frisbee Sailing"
> Copyright (C) 2013 The R Foundation for Statistical Computing
> Platform: x86_64-pc-linux-gnu (64-bit)
>
> R is free software and comes with ABSOLUTELY NO WARRANTY.
> You are welcome to redistribute it under certain conditions.
> Type 'license()' or 'licence()' for distribution details.
>
>   Natural language support but running in an English locale
>
> R is a collaborative project with many contributors.
> Type 'contributors()' for more information and
> 'citation()' on how to cite R or R packages in publications.
>
> Type 'demo()' for some demos, 'help()' for on-line help, or
> 'help.start()' for an HTML browser interface to help.
> Type 'q()' to quit R.
>
> [Previously saved workspace restored]
>
>> data=read.csv("output1.csv")
>> attach(data)
>> only_variables= subset(data, select=c(-date,-hour,-minute,-sec))
>>
>> library("vars")
> Loading required package: MASS
> Loading required package: strucchange
> Loading required package: zoo
>
> Attaching package: ‘zoo’
>
> The following objects are masked from ‘package:base’:
>
> as.Date, as.Date.numeric
>
> Loading required package: sandwich
> Loading required package: urca
> Loading required package: lmtest
>> summary(VAR(only_variables, p = 1, type ="both"))
> *Killed*
>
> [[alternative HTML version deleted]]
>
> __
> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] R getting "Killed" while running VAR model

2016-05-31 Thread Uwe Ligges 
Wild guess: You have huge and high dimensional VAR models, i.e. the 
matrices get huge and you use huge amounts of memory and you use more 
than what is available physically. The operating system protects itself 
by killing processes in such a case...


Best,
Uwe Ligges


On 31.05.2016 20:29, Vivek Singh wrote:

Hi,

I am using VARS (vector autoregressive model). The process gets killed
after running for sometime. Following is the output of R.

vivek@isds-research:~/cloudAuction/padding/panel$ cat var.Rout

R version 3.0.2 (2013-09-25) -- "Frisbee Sailing"
Copyright (C) 2013 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

[Previously saved workspace restored]


data=read.csv("output1.csv")
attach(data)
only_variables= subset(data, select=c(-date,-hour,-minute,-sec))

library("vars")

Loading required package: MASS
Loading required package: strucchange
Loading required package: zoo

Attaching package: ‘zoo’

The following objects are masked from ‘package:base’:

as.Date, as.Date.numeric

Loading required package: sandwich
Loading required package: urca
Loading required package: lmtest

summary(VAR(only_variables, p = 1, type ="both"))

*Killed*

[[alternative HTML version deleted]]

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.



__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

[R] R getting "Killed" while running VAR model

2016-05-31 Thread Vivek Singh 
Hi,

I am using VARS (vector autoregressive model). The process gets killed
after running for sometime. Following is the output of R.

vivek@isds-research:~/cloudAuction/padding/panel$ cat var.Rout

R version 3.0.2 (2013-09-25) -- "Frisbee Sailing"
Copyright (C) 2013 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

[Previously saved workspace restored]

> data=read.csv("output1.csv")
> attach(data)
> only_variables= subset(data, select=c(-date,-hour,-minute,-sec))
>
> library("vars")
Loading required package: MASS
Loading required package: strucchange
Loading required package: zoo

Attaching package: ‘zoo’

The following objects are masked from ‘package:base’:

as.Date, as.Date.numeric

Loading required package: sandwich
Loading required package: urca
Loading required package: lmtest
> summary(VAR(only_variables, p = 1, type ="both"))
*Killed*

[[alternative HTML version deleted]]

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.