The optim function is very resource hungry. I have had similar problems in
the past when dealing with extremely large datasets.
What is perhaps happening is that each 'step' of the optimization
algorithm stores some info so that it can compare to the next 'step', and
while the original vector may only be a few Mb of data, over many
iterations a huge amount memory is allocated to the optimization steps.
Maybe look at the control options under ?optim, particularly stuff like
trace, fnscale, ndeps, etc. that may cut down on the amount of data being
stored each step as well as the number of steps needed.
Good luck!
--------------------------------------
Jonathan P. Daily
Technician - USGS Leetown Science Center
11649 Leetown Road
Kearneysville WV, 25430
(304) 724-4480
"Is the room still a room when its empty? Does the room,
the thing itself have purpose? Or do we, what's the word... imbue it."
- Jubal Early, Firefly
From:
Lorenzo Cattarino <l.cattar...@uq.edu.au>
To:
David Winsemius <dwinsem...@comcast.net>, Peter Langfelder
<peter.langfel...@gmail.com>
Cc:
r-help@r-project.org
Date:
11/03/2010 03:26 AM
Subject:
Re: [R] memory allocation problem
Sent by:
r-help-boun...@r-project.org
Thanks for all your suggestions,
This is what I get after removing all the other (not useful) objects and
run my code:
> getsizes()
[,1]
org_results 47240832
myfun 11672
getsizes 4176
SS 3248
coeff 168
<NA> NA
<NA> NA
<NA> NA
<NA> NA
<NA> NA
> est_coeff <- optim(coeff,SS, steps=org_results$no.steps,
Range=org_results$Range, H1=org_results$H1, H2=org_results$H2,
p=org_results$p)
Error: cannot allocate vector of size 5.0 Mb
In addition: Warning messages:
1: In optim(coeff, SS, steps = org_results$no.steps, Range =
org_results$Range, :
Reached total allocation of 4055Mb: see help(memory.size)
2: In optim(coeff, SS, steps = org_results$no.steps, Range =
org_results$Range, :
Reached total allocation of 4055Mb: see help(memory.size)
3: In optim(coeff, SS, steps = org_results$no.steps, Range =
org_results$Range, :
Reached total allocation of 4055Mb: see help(memory.size)
4: In optim(coeff, SS, steps = org_results$no.steps, Range =
org_results$Range, :
Reached total allocation of 4055Mb: see help(memory.size)
>
It seems that R is using all the default availabe memory (4 GB, which is
the RAM of my processor).
> memory.limit()
[1] 4055
> memory.size()
[1] 4049.07
>
My dataframe has a size of 47240832 bytes, or about 45 Mb. So it should
not be a problem in terms of memory usage?
I do not understand what is going on.
Thanks for your help anyway
Lorenzo
-----Original Message-----
From: David Winsemius [mailto:dwinsem...@comcast.net]
Sent: Wednesday, 3 November 2010 12:48 PM
To: Lorenzo Cattarino
Cc: r-help@r-project.org
Subject: Re: [R] memory allocation problem
Restart your computer. (Yeah, I know that what the help-desk always
says.)
Start R before doing anything else.
Then run your code in a clean session. Check ls() oafter starte up to
make sure you don't have a bunch f useless stuff in your .Rdata
file. Don't load anything that is not germane to this problem. Use
this function to see what sort of space issues you might have after
loading objects:
getsizes <- function() {z <- sapply(ls(envir=globalenv()),
function(x) object.size(get(x)))
(tmp <- as.matrix(rev(sort(z))[1:10]))}
Then run your code.
--
David.
On Nov 2, 2010, at 10:13 PM, Lorenzo Cattarino wrote:
> I would also like to include details on my R version
>
>
>
>> version _
>
> platform x86_64-pc-mingw32
> arch x86_64
>
> os mingw32
> system x86_64, mingw32
> status
> major 2
> minor 11.1
> year 2010
> month 05
> day 31
> svn rev 52157
> language R
> version.string R version 2.11.1 (2010-05-31)
>
> from FAQ 2.9
>
(http://cran.r-project.org/bin/windows/base/rw-FAQ.html#There-seems-to-b
> e-a-limit-on-the-memory-it-uses_0021
>
<http://cran.r-project.org/bin/windows/base/rw-FAQ.html#There-seems-to-b
> e-a-limit-on-the-memory-it-uses_0021> ) it says that:
> "For a 64-bit build, the default is the amount of RAM"
>
> So in my case the amount of RAM would be 4 GB. R should be able to
> allocate a vector of size 5 Mb without me typing any command (either
> as
> memory.limit() or appended string in the target path), is that right?
>
>
>
> From: Lorenzo Cattarino
> Sent: Wednesday, 3 November 2010 10:55 AM
> To: 'r-help@r-project.org'
> Subject: memory allocation problem
>
>
>
> I forgot to mention that I am using windows 7 (64-bit) and the R
> version
> 2.11.1 (64-bit)
>
>
>
> From: Lorenzo Cattarino
>
> I am trying to run a non linear parameter optimization using the
> function optim() and I have problems regarding memory allocation.
>
> My data are in a dataframe with 9 columns. There are 656100 rows.
>
>> head(org_results)
>
> comb.id p H1 H2 Range Rep no.steps dist aver.hab.amount
>
> 1 1 0.1 0 0 1 100 0
> 0.2528321 0.1393901
>
> 2 1 0.1 0 0 1 100 0
> 0.4605934 0.1011841
>
> 3 1 0.1 0 0 1 100 4
> 3.4273670 0.1052789
>
> 4 1 0.1 0 0 1 100 4
> 2.8766364 0.1022138
>
> 5 1 0.1 0 0 1 100 0
> 0.3496872 0.1041056
>
> 6 1 0.1 0 0 1 100 0
> 0.1050840 0.3572036
>
>> est_coeff <- optim(coeff,SS, steps=org_results$no.steps,
> Range=org_results$Range, H1=org_results$H1, H2=org_results$H2,
> p=org_results$p)
>
> Error: cannot allocate vector of size 5.0 Mb
>
> In addition: Warning messages:
>
> 1: In optim(coeff, SS, steps = org_results$no.steps, Range =
> org_results$Range, : Reached total allocation of 10000Mb: see
> help(memory.size)
>
> 2: In optim(coeff, SS, steps = org_results$no.steps, Range =
> org_results$Range, : Reached total allocation of 10000Mb: see
> help(memory.size)
>
> 3: In optim(coeff, SS, steps = org_results$no.steps, Range =
> org_results$Range, : Reached total allocation of 10000Mb: see
> help(memory.size)
>
> 4: In optim(coeff, SS, steps = org_results$no.steps, Range =
> org_results$Range, : Reached total allocation of 10000Mb: see
> help(memory.size)
>
>> memory.size()
>
> [1] 9978.19
>
>> memory.limit()
>
> [1] 10000
>
>>
>
> I know that I am not sending reproducible codes but I was hoping that
> you could help me understand what is going on. I set a maximum limit
> of
> 10000 mega byte (by writing this string --max-mem-size=10000M after
> the
> target path, right click on R icon, shortcut tab). And R is telling me
> that it cannot allocate a vector of size 5 Mb???
>
>
David Winsemius, MD
West Hartford, CT
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
[[alternative HTML version deleted]]
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
