You may be interested in this thread:
https://www.stat.math.ethz.ch/pipermail/r-help/2004-June/052102.html
-roger
HENRIKSON, JEFFREY wrote:
Hello,
I have trouble using read.table for flat files of larger than about 300MB on windows 2000. Any ideas of how to file a bug report? Is it a known issue? I have three cuts of data, a 1%, 10% and 100% sample in flat text files. The 100% sample is about 350MB. When I read the 1% and 10% files, besides being slow, everything works. RAM footprint appears to increase approximately 2x of text file size when loaded. I have 1.5GB of ram on my machine. The 10% file takes < 1.5 minutes to load. So the 100% file I would think would load in < 15 minutes. But it grinds for about 15 mins and then seg faults instead. I don't think there's really very special about my data. Just several columns by ~5M rows.
The same thing happens when I read the 100% sample in from an RDBMS with RODBC.
For the time being I have worked around by feeding in small cross sections 100% from the RDBMS, and storing a 10% whole sample in RAM. But in the future it would be nice if I could just use the RAM in my box.
Jeff Henrikson
______________________________________________ [EMAIL PROTECTED] mailing list https://www.stat.math.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
-- Roger D. Peng http://www.biostat.jhsph.edu/~rpeng/
______________________________________________ [EMAIL PROTECTED] mailing list https://www.stat.math.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
