* William Dunlap jqha...@gvopb.pbz [2012-02-28 23:06:54 +]:
You need to walk through the objects, checking for environments on
each component or attribute of an object.
so why doesn't object.size do that?
f - function(n) {
+ d - data.frame(y = rnorm(n), x = rnorm(n))
+ lm(y
I do a lot of strsplit, unlist, subsetting, so I could imagine why
the RSS is triple the total size of my data if all the intermediate
results are not released.
I can only give some generalities about that. Using lots of
small chunks of memory (like short strings) may cause fragmentation
Le mercredi 29 février 2012 à 11:42 -0500, Sam Steingold a écrit :
* William Dunlap jqha...@gvopb.pbz [2012-02-28 23:06:54 +]:
You need to walk through the objects, checking for environments on
each component or attribute of an object.
so why doesn't object.size do that?
f -
* Milan Bouchet-Valat anyvzv...@pyho.se [2012-02-29 18:18:50 +0100]:
I think you're simply hitting a (terrible) OS limitation. Linux is
very often not able to reclaim the memory R has used because it's
fragmented. The OS can only get the pages back if nothing is above
them, and most of the
On Wed, 29 Feb 2012, Sam Steingold wrote:
* Milan Bouchet-Valat anyvzv...@pyho.se [2012-02-29 18:18:50 +0100]:
I think you're simply hitting a (terrible) OS limitation. Linux is
very often not able to reclaim the memory R has used because it's
fragmented. The OS can only get the pages back if
* yhxr-gvre...@hvbjn.rqh [2012-02-29 13:55:25 -0600]:
On Wed, 29 Feb 2012, Sam Steingold wrote:
compacting garbage collector is our best friend!
Which R does not use because of the problems it would create for
external C/Fortran code on which R heavily relies.
Well, you know better, of
My basic worry is that the GC does not work properly,
i.e., the unreachable data is never collected.
* Bert Gunter thagre.ore...@trar.pbz [2012-02-27 14:35:14 -0800]:
This appears to be the sort of query that (with apologies to other R
gurus) only Brian Ripley or Luke Tierney could figure
On Tue, Feb 28, 2012 at 11:57 AM, Sam Steingold s...@gnu.org wrote:
My basic worry is that the GC does not work properly,
i.e., the unreachable data is never collected.
Highly unlikely. Such basic inner R code has been well tested over 20
years. I believe that you merely don't understand the
-project.org] On
Behalf Of Sam Steingold
Sent: Tuesday, February 28, 2012 11:58 AM
To: r-help@r-project.org; Bert Gunter
Subject: Re: [R] memory management
My basic worry is that the GC does not work properly,
i.e., the unreachable data is never collected.
* Bert Gunter thagre.ore...@trar.pbz
* William Dunlap jqha...@gvopb.pbz [2012-02-28 20:19:06 +]:
Look into environments that may be stored with your data.
thanks, but I see nothing like that:
for (n in ls(all.names = TRUE)) {
o - get(n)
print(object.size(o), units=Kb)
e - environment(o)
if (!identical(e,NULL)
You need to walk through the objects, checking for
environments on each component or attribute of an
object. You also have to look at the parent.env
of each environment found. E.g.,
f - function(n) {
+ d - data.frame(y = rnorm(n), x = rnorm(n))
+ lm(y ~ poly(x, 4), data=d)
+ }
z
It appears that the intermediate data in functions is never GCed even
after the return from the function call.
R's RSS is 4 Gb (after a gc()) and
sum(unlist(lapply(lapply(ls(),get),object.size)))
[1] 1009496520
(less than 1 GB)
how do I figure out where the 3GB of uncollected garbage is hiding?
This appears to be the sort of query that (with apologies to other R
gurus) only Brian Ripley or Luke Tierney could figure out. R generally
passes by value into function calls (but not *always*), so often
multiple copies of objects are made during the course of calls. I
would speculate that this
zz - data.frame(a=c(1,2,3),b=c(4,5,6))
zz
a b
1 1 4
2 2 5
3 3 6
a - zz$a
a
[1] 1 2 3
a[2] - 100
a
[1] 1 100 3
zz
a b
1 1 4
2 2 5
3 3 6
clearly a is a _copy_ of its namesake column in zz.
when was the copy made? when a was modified? at assignment?
is there a way to find out how
This should help:
invisible(gc())
m0 - memory.size()
mem.usage - function(){invisible(gc()); memory.size() - m0}
Mb.size - function(x)print(object.size(x), units=Mb)
zz - data.frame(a=runif(100), b=runif(100))
mem.usage()
[1] 15.26
Mb.size(zz)
15.3 Mb
a - zz$a
mem.usage()
[1]
* Florent D. syb...@tznvy.pbz [2012-02-09 19:26:59 -0500]:
m0 - memory.size()
Mb.size - function(x)print(object.size(x), units=Mb)
indeed, these are very useful, thanks.
ls reports these objects larger than 100k:
behavior : 390.1 Mb
mydf : 115.3 Mb
nb : 0.2 Mb
pl : 1.2 Mb
however, top
I am trying to run a very large Bradley-Terry model using the BradleyTerry2
package. (There are 288 players in the BT model).
My problem is that I ran the model below successfully.
WLMat is a win-loss matrix that is 288 by 288
WLdf-countsToBinomial(WLMat)
It would be very useful if you would post some information about what
exactly you are doing. There si something with the size of the data
object you are processing ('str' would help us understand it) and then
a portion of the script (both before and after the error message) so
we can understand
jholt...@gmail.com
Assunto: Re: [R] Memory Management under Linux
Para: ricardo souza ricsouz...@yahoo.com.br
Cc: r-help@r-project.org
Data: Sexta-feira, 5 de Novembro de 2010, 10:21
It would be very useful if you would post some information about what
exactly you are doing. There si something
,
Ricardo
De: jim holtman jholt...@gmail.com
Assunto: Re: [R] Memory Management under Linux
Para: ricardo souza ricsouz...@yahoo.com.br
Cc: r-help@r-project.org
Data: Sexta-feira, 5 de Novembro de 2010, 10:21
It would be very useful if you would post some information about what
exactly
Dear all,
I am using ubuntu linux 32 with 4 Gb. I am running a very small script and I
always got the same error message: CAN NOT ALLOCATE A VECTOR OF SIZE 231.8 Mb.
I have reading carefully the instruction in ?Memory. Using the function gc() I
got very low numbers of memory (please sea
I already offered the Biostrings package. It provides more robust
methods for string matching than does grepl. Is there a reason that you
choose not to?
Indeed that is the way I should go for and I have installed the package
after some struggling. Since biostring is a fairly complex package
Date: Sun, 10 Oct 2010 15:27:11 +0200
From: lorenzo.ise...@gmail.com
To: dwinsem...@comcast.net
CC: r-help@r-project.org
Subject: Re: [R] Memory management in R
I already offered the Biostrings package. It provides more robust
methods
Hi David,
I am replying to you and to the other people who provided some insight
into my problems with grepl.
Well, at least we now know that the bug is reproducible.
Indeed it is a strange sequence the one I am postprocessing, probably
pathological to some extent, nevertheless the problem is
On Oct 9, 2010, at 9:45 AM, Lorenzo Isella wrote:
Hi David,
I am replying to you and to the other people who provided some
insight into my problems with grepl.
Well, at least we now know that the bug is reproducible.
Indeed it is a strange sequence the one I am postprocessing,
probably
My suggestion is to explore other alternatives. (I will admit that I
don't yet fully understand the test that you are applying.)
Hi,
I am trying to partially implement the Lempel Ziv compression algorithm.
The point is that compressibility and entropy of a time series are
related, hence my
On Oct 9, 2010, at 4:23 PM, Lorenzo Isella wrote:
My suggestion is to explore other alternatives. (I will admit that I
don't yet fully understand the test that you are applying.)
Hi,
I am trying to partially implement the Lempel Ziv compression
algorithm.
The point is that
is some
limitation of grepl or R memory management.
Any idea about how I could tackle this problem or how I can profile my
code to fix it (though it really seems to me that I have to find a way
to allow R to process longer strings).
Any suggestion is appreciated.
Cheers
Lorenzo
-boun...@r-project.org]
Sent: Friday, October 08, 2010 1:12 PM
To: r-help
Subject: [R] Memory management in R
Dear All,
I am experiencing some problems with a script of mine.
It crashes with this message
Error in grepl(fut_string, past_string) :
invalid regular expression
'12653a6#12653a6
the problem is some
limitation of grepl or R memory management.
Any idea about how I could tackle this problem or how I can profile my
code to fix it (though it really seems to me that I have to find a way
to allow R to process longer strings).
Any suggestion is appreciated.
Cheers
Lorenzo
) and the problem does not seem to be RAM memory (I have
several GB of RAM on my machine and its consumption never shoots up so my
machine never resorts to swap memory).
So (though I am not an expert) it looks like the problem is some limitation
of grepl or R memory management.
Any idea about how I could
Date: Fri, 8 Oct 2010 13:30:59 -0400
From: jholt...@gmail.com
To: lorenzo.ise...@gmail.com
CC: r-help@r-project.org
Subject: Re: [R] Memory management in R
More specificity: how long is the string, what is the pattern you are
matching against
like the problem is some limitation
of grepl or R memory management.
Any idea about how I could tackle this problem or how I can profile my code
to fix it (though it really seems to me that I have to find a way to allow R
to process longer strings).
Any suggestion is appreciated.
Cheers
Lorenzo
and its consumption never shoots
up so my
machine never resorts to swap memory).
So (though I am not an expert) it looks like the problem is some
limitation
of grepl or R memory management.
Any idea about how I could tackle this problem or how I can
profile my code
to fix it (though
From: dwinsem...@comcast.net
To: lorenzo.ise...@gmail.com
Date: Fri, 8 Oct 2010 19:30:45 -0400
CC: r-help@r-project.org
Subject: Re: [R] Memory management in R
On Oct 8, 2010, at 6:42 PM, Lorenzo Isella wrote:
Please find below the R
On Oct 8, 2010, at 9:19 PM, Mike Marchywka wrote:
From: dwinsem...@comcast.net
To: lorenzo.ise...@gmail.com
Date: Fri, 8 Oct 2010 19:30:45 -0400
CC: r-help@r-project.org
Subject: Re: [R] Memory management in R
On Oct 8, 2010, at 6:42 PM, Lorenzo
I have volunteered to give a short talk on memory management in R
to my local R user group, mainly to motivate myself to learn about it.
The focus will be on what a typical R coder might want to know ( e.g. how
objects are created, call by value, basics of garbage collection ) but I
want
, short integers etc.).
Jens Oehlschlägel
-Ursprüngliche Nachricht-
Von: john mull...@fastmail.fm
Gesendet: Jun 16, 2010 12:20:17 PM
An: r-help@r-project.org
Betreff: [R] memory management in R
I have volunteered to give a short talk on memory management in R
to my local R user group
I'm wondering where I can find the detailed descriptions on R memory
management. Understanding this could help me understand the runtime of
R program. For example, depending on how memory is allocated (either
allocate a chuck of memory that is more than necessary for the current
use, or allocate
Related...
Rule of thumb:
Pre-allocate your object of the *correct* data type, if you know the
final dimensions.
/Henrik
On Thu, Dec 10, 2009 at 8:26 AM, Peng Yu pengyu...@gmail.com wrote:
I'm wondering where I can find the detailed descriptions on R memory
management. Understanding
pengyu...@gmail.com wrote:
I'm wondering where I can find the detailed descriptions on R memory
management. Understanding this could help me understand the runtime of
R program. For example, depending on how memory is allocated (either
allocate a chuck of memory that is more than necessary
:
Related...
Rule of thumb:
Pre-allocate your object of the *correct* data type, if you know the
final dimensions.
/Henrik
On Thu, Dec 10, 2009 at 8:26 AM, Peng Yu pengyu...@gmail.com wrote:
I'm wondering where I can find the detailed descriptions on R memory
management. Understanding this could
Yu pengyu...@gmail.com wrote:
I'm wondering where I can find the detailed descriptions on R memory
management. Understanding this could help me understand the runtime of
R program. For example, depending on how memory is allocated (either
allocate a chuck of memory that is more than
wrote:
Related...
Rule of thumb:
Pre-allocate your object of the *correct* data type, if you know the
final dimensions.
/Henrik
On Thu, Dec 10, 2009 at 8:26 AM, Peng Yu pengyu...@gmail.com wrote:
I'm wondering where I can find the detailed descriptions on R memory
management
Hi,
I'm using R to collect data for a number of exchanges through a socket
connection and constantly running into memory problems even though task I
believe is not that memory consuming. I guess there is a miscommunication
between R and WinXP about freeing up memory.
So this is the code:
The line:
data. - c(data., new.data)
will eat both memory and time voraciously.
You should change it by creating 'data.' to
be the final size it will be and then subscript
into it. If you don't know the final size, then
you can grow it a lot a few times instead of
growing it a little lots of
Hi,
I apologize again for posting something not suitable on this list.
Basically, it sounds like I should go put this large dataset into a
database... The dataset I have had trouble with is the transportation
network of Chicago Consolidated Metropolitan Statistical Area. The
number of samples
If you data file has 49M rows and 249 columns, then if each column had
5 characters, then you are looking at a text file with 60GB. If these
were all numerics (8 bytes per number), then you are looking at an R
object that would be almost 100GB. If this is your data, then this is
definitely a
48 matches
Mail list logo
