There seem to have been three almost identical reports of this from three
different email addresses, including two widely-separate UK academic
institutions, and two user names, which is a bit weird.
However: R will need more than 4GB memory to handle a 3.99GB file, and
probably more than 4GB memory to handle a 1.33GB file.
On 32-bit R I think the recommendation is that the data should occupy no more
than 10% of the address space, ie, about 400MB. For 64-bit R, you are likely
to still find R very slow if the data set is more than about 1/3 of physical
memory. For a 4GB data file I would recommend a computer with at least 16GB
memory.
Now, it is often possible to load only a small fraction of the data at one time
and so to analyse large data sets on smaller computers. Three examples that I
have worked on
- analysing data from a survey data set about the same size as your data,
by keeping most of the data in a SQLite database and just loading a
few variables at a time.
- fitting linear regression models to large data sets with the biglm
package, by keeping the data in a SQLite database and just loading a few rows
at a time
- analysing whole-genome genetic data (40GB or so) by storing the data in a
netCDF file and reading appropriate chunks with the ncdf package.
-thomas
On Wed, 17 Mar 2010, Mamun wrote:
Dear List,
I am trying to read some files using read.csv and total size of those files
are 3.99 GB. I am using MacBook Pro with 4GB RAM(snow leopard). I also tried
to run a chunk from those files and altogether the size was 1.33 GB. But
every time I was getting the following error
R(1200) malloc: *** mmap(size=16777216) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
I managed to run smaller chunks (400MB) and I also saved the object. But
problem is when I try to load some of those objects(which is not more than
1.5 GB altogether), I get same error again. Can someone please help? why am
I getting this error? does R need more space than the actual file size? I
may buy new machine if its something related to RAM size. but if it is some
R problem then it will be very useless to buy new machine. So, I really need
to know why this is happening.
Any help is appreciated.
Thanks in advance.
regards,
Mamun
--
View this message in context:
http://n4.nabble.com/mmap-error-12-macbook-pro-tp1596234p1596234.html
Sent from the R help mailing list archive at Nabble.com.
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Thomas Lumley Assoc. Professor, Biostatistics
tlum...@u.washington.edu University of Washington, Seattle
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
