Re: [R] Error: cannot allocate vector of size x Gb (64-bit ... yet again)
On Tue, 23 Nov 2010, derek eder wrote: Hello, I am facing the dreaded Error: cannot allocate vector of size x Gb and don't understand enough about R (or operating system) memory management to diagnose and solve the problem -- despite studying previous posts and relevant R help -- e.g.: Error messages beginning cannot allocate vector of size indicate a failure to obtain memory, either because the size exceeded the address-space limit for a process or, more likely, because the system was unable to provide the memory. [...] On all builds of R, the maximum length (number of elements) of a vector is 2^31 - 1 ~ 2*10^9, as lengths are stored as signed integers. In addition, the storage space cannot exceed the address limit. - from Memory-limits {Base} Simple question: Given 64-bit R (AMD64 Linux) with a ulimit of unlimited, can the size of an R object exceed the amount of availlable RAM memory? Empirically my system with 4Gb RAM and ample Swap, is failing: x - integer(10^9) object.size(x) 400040 bytes gc() used (Mb) gc trigger (Mb) max used (Mb) Ncells1211956.5 35 18.735 18.7 Vcells 500124024 3815.7 606849099 4629.9 550124408 4197.2 matrix(x, ncol=16) Error: cannot allocate vector of size 3.7 Gb I don't understand how this operation violates the limits detailed in the Memory-limit help (above). It doesn't, and that example works on my (32GB RAM) 64-bit system gc() used (Mb) gc trigger(Mb) max used(Mb) Ncells 1340547.2 3518.7 3518.7 Vcells 1000131158 7630.4 1575382823 12019.3 1500131550 11445.1 So maybe you don't have 'ample swap' and/or have a VM limit set. Note that dim(x) - c(length(x)/16, 16) is much more efficient use of memory. Thank you! Derek Eder - version _ platform x86_64-pc-linux-gnu arch x86_64 os linux-gnu system x86_64, linux-gnu status major 2 minor 11.1 year 2010 month 05 day31 svn rev52157 language R version.string R version 2.11.1 (2010-05-31) de...@papanca:~$ top top - 09:10:18 up 51 min, 4 users, load average: 0.51, 0.51, 0.45 Tasks: 160 total, 2 running, 158 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 25.0%sy, 0.0%ni, 75.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 3796484k total, 3764852k used,31632k free,14204k buffers Swap: 2929660k total, 834240k used, 2095420k free,94800k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 2854 derek 20 0 239m 9260 5448 S6 0.2 0:05.53 gnome-terminal 1164 root 20 0 218m 31m 10m S4 0.8 1:29.71 Xorg 3331 derek 20 0 19276 1324 944 R1 0.0 0:00.6 top __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- Brian D. Ripley, rip...@stats.ox.ac.uk Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/ University of Oxford, Tel: +44 1865 272861 (self) 1 South Parks Road, +44 1865 272866 (PA) Oxford OX1 3TG, UKFax: +44 1865 272595 __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] Error: cannot allocate vector of size x Gb (64-bit ... yet again)
Hello, I am facing the dreaded Error: cannot allocate vector of size x Gb and don't understand enough about R (or operating system) memory management to diagnose and solve the problem -- despite studying previous posts and relevant R help -- e.g.: Error messages beginning cannot allocate vector of size indicate a failure to obtain memory, either because the size exceeded the address-space limit for a process or, more likely, because the system was unable to provide the memory. [...] On all builds of R, the maximum length (number of elements) of a vector is 2^31 - 1 ~ 2*10^9, as lengths are stored as signed integers. In addition, the storage space cannot exceed the address limit. - from Memory-limits {Base} Simple question: Given 64-bit R (AMD64 Linux) with a ulimit of unlimited, can the size of an R object exceed the amount of availlable RAM memory? Empirically my system with 4Gb RAM and ample Swap, is failing: x - integer(10^9) object.size(x) 400040 bytes gc() used (Mb) gc trigger (Mb) max used (Mb) Ncells1211956.5 35 18.735 18.7 Vcells 500124024 3815.7 606849099 4629.9 550124408 4197.2 matrix(x, ncol=16) Error: cannot allocate vector of size 3.7 Gb I don't understand how this operation violates the limits detailed in the Memory-limit help (above). Thank you! Derek Eder - version _ platform x86_64-pc-linux-gnu arch x86_64 os linux-gnu system x86_64, linux-gnu status major 2 minor 11.1 year 2010 month 05 day31 svn rev52157 language R version.string R version 2.11.1 (2010-05-31) de...@papanca:~$ top top - 09:10:18 up 51 min, 4 users, load average: 0.51, 0.51, 0.45 Tasks: 160 total, 2 running, 158 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 25.0%sy, 0.0%ni, 75.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 3796484k total, 3764852k used,31632k free,14204k buffers Swap: 2929660k total, 834240k used, 2095420k free,94800k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 2854 derek 20 0 239m 9260 5448 S6 0.2 0:05.53 gnome-terminal 1164 root 20 0 218m 31m 10m S4 0.8 1:29.71 Xorg 3331 derek 20 0 19276 1324 944 R1 0.0 0:00.6 top __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Error: cannot allocate vector of size x Gb (64-bit ... yet again)
On 23.11.2010 09:26, derek eder wrote: Hello, I am facing the dreaded Error: cannot allocate vector of size x Gb and don't understand enough about R (or operating system) memory management to diagnose and solve the problem -- despite studying previous posts and relevant R help -- e.g.: Error messages beginning cannot allocate vector of size indicate a failure to obtain memory, either because the size exceeded the address-space limit for a process or, more likely, because the system was unable to provide the memory. [...] On all builds of R, the maximum length (number of elements) of a vector is 2^31 - 1 ~ 2*10^9, as lengths are stored as signed integers. In addition, the storage space cannot exceed the address limit. - from Memory-limits {Base} Simple question: Given 64-bit R (AMD64 Linux) with a ulimit of unlimited, can the size of an R object exceed the amount of availlable RAM memory? Empirically my system with 4Gb RAM and ample Swap, is failing: x - integer(10^9) object.size(x) 400040 bytes gc() used (Mb) gc trigger (Mb) max used (Mb) Ncells 121195 6.5 35 18.7 35 18.7 Vcells 500124024 3815.7 606849099 4629.9 550124408 4197.2 matrix(x, ncol=16) Error: cannot allocate vector of size 3.7 Gb I don't understand how this operation violates the limits detailed in the Memory-limit help (above). Why? If you have a limit of 4Gb, 4Gb are already used and you try to work with a 4Gb objeczt in the matrix function which makes a copy and creates another copy (with dim attributed), so 8Gb additionally. You will need at least 12 of your 4Gb. Probably R does not get a sufficient amount of non-fragmented memory from the OS or your swap space is not ample. To make the above more efficient, just use: dim(x) - c(10^8/16, 16) and you won't get any copies. Uwe Thank you! Derek Eder - version _ platform x86_64-pc-linux-gnu arch x86_64 os linux-gnu system x86_64, linux-gnu status major 2 minor 11.1 year 2010 month 05 day 31 svn rev 52157 language R version.string R version 2.11.1 (2010-05-31) de...@papanca:~$ top top - 09:10:18 up 51 min, 4 users, load average: 0.51, 0.51, 0.45 Tasks: 160 total, 2 running, 158 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 25.0%sy, 0.0%ni, 75.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 3796484k total, 3764852k used, 31632k free, 14204k buffers Swap: 2929660k total, 834240k used, 2095420k free, 94800k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2854 derek 20 0 239m 9260 5448 S 6 0.2 0:05.53 gnome-terminal 1164 root 20 0 218m 31m 10m S 4 0.8 1:29.71 Xorg 3331 derek 20 0 19276 1324 944 R 1 0.0 0:00.6 top __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.