[R] Error: cannot allocate vector of size x Gb (64-bit ... yet again)
Hello,
I am facing the dreaded Error: cannot allocate vector of size x Gb and
don't understand
enough about R (or operating system) memory management to diagnose and
solve the problem
-- despite studying previous posts and relevant R help -- e.g.:
Error messages beginning cannot allocate vector of size indicate a
failure to obtain memory,
either because the size exceeded the address-space limit for a process
or, more likely,
because the system was unable to provide the memory.
[...] On all builds of R, the maximum length (number of elements)
of a vector is 2^31 - 1 ~ 2*10^9, as lengths are stored as signed integers.
In addition, the storage space cannot exceed the address limit.
- from Memory-limits {Base}
Simple question: Given 64-bit R (AMD64 Linux) with a ulimit of
unlimited, can the size of an R object exceed the amount of availlable RAM
memory?
Empirically my system with 4Gb RAM and ample Swap, is failing:
x - integer(10^9)
object.size(x)
400040 bytes
gc()
used (Mb) gc trigger (Mb) max used (Mb)
Ncells1211956.5 35 18.735 18.7
Vcells 500124024 3815.7 606849099 4629.9 550124408 4197.2
matrix(x, ncol=16)
Error: cannot allocate vector of size 3.7 Gb
I don't understand how this operation violates the limits detailed in
the Memory-limit help (above).
Thank you!
Derek Eder
-
version
_
platform x86_64-pc-linux-gnu
arch x86_64
os linux-gnu
system x86_64, linux-gnu
status
major 2
minor 11.1
year 2010
month 05
day31
svn rev52157
language R
version.string R version 2.11.1 (2010-05-31)
de...@papanca:~$ top
top - 09:10:18 up 51 min, 4 users, load average: 0.51, 0.51, 0.45
Tasks: 160 total, 2 running, 158 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 25.0%sy, 0.0%ni, 75.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Mem: 3796484k total, 3764852k used,31632k free,14204k buffers
Swap: 2929660k total, 834240k used, 2095420k free,94800k cached
PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
2854 derek 20 0 239m 9260 5448 S6 0.2 0:05.53
gnome-terminal
1164 root 20 0 218m 31m 10m S4 0.8 1:29.71 Xorg
3331 derek 20 0 19276 1324 944 R1 0.0 0:00.6 top
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
[R] calculating treatment effects (differences) in a data frame?
I am trying to calculating the treatment effect for individual subjects (ID) of a (score) between 2 time-points (visit) (see example below). The data is in an unbalanced data.frame in long format with some missing data. I suspect that I am overlooking a very simple function, something along the lines of tapply(). Thank you for you attention! Derek Eder ## Examples: myData = data.frame( ID = c(a,a,b,c,c,d,d), visit=c(1,2,1,1,2,1,2), score=c(10,2,12,16,0,NA,5) ) myData ID visit score 1 a 110 2 a 2 2 3 b 112 4 c 116 5 c 2 0 6 d 1NA 7 d 2 5 # The desired result is a vector of time differences by ID # a b c d # 8 NA 16 NA ## solutions ? # This works, but the returned data frame is awkward for me # because the empty cells (b and d) contain integer(0) # and not the more familiar NA. aggregate(data=myData, score~ID,FUN=diff) ID score 1 a-8 2 b 3 c -16 4 d # This works as desired ... but somehow seems unecessarily complicated reshape(data=myData,timevar=visit,idvar=ID, direction=wide) ID score.1 score.2 1 a 10 2 3 b 12 NA 4 c 16 0 6 d NA 5 apply(X = reshape(data=myData,timevar=visit,idvar=ID, direction=wide)[,-1], MARGIN = 1, FUN = diff) 1 3 4 6 -8 NA -16 NA __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] Saving interactive modified latttice plots?
The lattice function panel.identify() allows the labeling of mouse-selected plot points. But ... how does one save the resulting plot? E.g., to png() or pdf() Thank you! # example code library(lattice) xyplot(rnorm(10,0,1)~rnorm(10,0,1)) trellis.focus() # left click on plot panel to select panel.identify() # left click on selected points. Right click to exit process trellis.unfocus() Derek N. Eder Gothenburg University The most dangerous thing in the jungle is not the snakes, the spiders, the tigers, ... The most dangerous thing in the jungle is your mind! __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] Converting from date strings to internal POSIX format?
I need to convert a date-style string: 2009-06-16 09:28:17.746
To its POSIX representation: 1245137297746
The function below converts my POSIX date to a string ... now I need to go
backwards!
render.t32 - function(t32, tz = CET)
{
timez - ISOdatetime(1970,1,1,0,0,0, tz =UTC)+t32/1000
return(format(timez,%Y-%m-%d %H:%M:%OS3,tz=CET))
}
render.t32(1245137297746)
[1] 2009-06-16 09:28:17.746
Have not succeeded in my attempts to understand POSIX
and I thank you for your attention.
~Derek
Derek N. Eder
Gothenburg University
Vigilance and Neurocognition Laboratory
Medicinaregatan 8B
Box 421
Gothenburg Sweden
SE 405 30
tlf (031) 342-8261
mobil 0704 915 714
The most dangerous thing in the jungle is not the snakes, the spiders, the
tigers, ...
The most dangerous thing in the jungle is your mind!
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
[R] interactive file choosing in Linux?
I am used to using the [R] function choose.files() for interactive file selection in MS-Windows. What is the comparable function in Linux? I expected the function file.choose() to display similar behavior, i.e., a graphical interface diplaying a file listing, but all I seem to get is a text input prompt. This does not seem correct. file.choose() Enter file name: I have seen gfile() function in the gWidgets library - but isn't there anything native? Thank you! Derek Eder Linux: Ubuntu 9.04, Gnome, [R] running in terminal or ESS GTK_Emacs (doesn't make any difference to the above). version _ platform i486-pc-linux-gnu arch i486 os linux-gnu system i486, linux-gnu status major 2 minor 8.1 year 2008 month 12 day 22 svn rev 47281 language R version.string R version 2.8.1 (2008-12-22) Derek N. Eder Gothenburg University Vigilance and Neurocognition Laboratory Medicinaregatan 8B Gothenburg Sweden SE 405 30 tlf (031) 342-8261 mobil 0704 915 714 All created things are impermanent — Strive diligently. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
