[R] problem in reading large files

2006-08-11 Thread T Mu
I was trying to read a large .csv file (80 colums, 400,000 rows, size of
about 200MB). I used scan(), R 2.3.1 on Windows XP. My computer is AMD 2000+
and has 512MB ram.

It sometimes freezes my PC, sometimes just shuts down R quitely.

Is there a way (option, function) to better handle large files?

Seemingly SAS can deal with it with no problem, but I just persuaded my
professor transfering to R, so it is quite disappointing.

Please help, thank you.

[[alternative HTML version deleted]]

__
R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.


Re: [R] problem in reading large files

2006-08-11 Thread jim holtman
You might consider processing this is pieces.  If I assume that this is
numeric data, then 'after' conversion this will take up 256MB of RAM which
is 1/2 your real memory. This means that you will probably not be able to do
any processing of the data since you will probably have to make copies of
parts of the object at various times.  It also assumes that you have all the
512MB really available to your program; there is the operating system and
other applications running.

Do you have to have all the data in memory at one time?  If so, consider
getting a 64-bit version of R and at least 2GB of RAM to be on the safe
side.  You could also put the data in a database and then read in only the
columns that are needed.


On 8/11/06, T Mu [EMAIL PROTECTED] wrote:

 I was trying to read a large .csv file (80 colums, 400,000 rows, size of
 about 200MB). I used scan(), R 2.3.1 on Windows XP. My computer is AMD
 2000+
 and has 512MB ram.

 It sometimes freezes my PC, sometimes just shuts down R quitely.

 Is there a way (option, function) to better handle large files?

 Seemingly SAS can deal with it with no problem, but I just persuaded my
 professor transfering to R, so it is quite disappointing.

 Please help, thank you.

[[alternative HTML version deleted]]

 __
 R-help@stat.math.ethz.ch mailing list
 https://stat.ethz.ch/mailman/listinfo/r-help
 PLEASE do read the posting guide
 http://www.R-project.org/posting-guide.html
 and provide commented, minimal, self-contained, reproducible code.




-- 
Jim Holtman
Cincinnati, OH
+1 513 646 9390

What is the problem you are trying to solve?

[[alternative HTML version deleted]]

__
R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.


Re: [R] problem in reading large files

2006-08-11 Thread Duncan Murdoch
On 8/11/2006 8:50 PM, T Mu wrote:
 I was trying to read a large .csv file (80 colums, 400,000 rows, size of
 about 200MB). I used scan(), R 2.3.1 on Windows XP. My computer is AMD 2000+
 and has 512MB ram.

You should get R-patched; there were some bugs with low memory handling 
fixed recently:

 From CHANGES:

R could crash when very low on memory. (PR#8981)

You should also get more physical memory.  512MB is not much for 
handling a 200MB of data.  You can fairly easily benefit from increasing 
up to 2 GB, and will benefit (with some work) if you have even more, up 
to 4 GB.

Duncan Murdoch

 
 It sometimes freezes my PC, sometimes just shuts down R quitely.
 
 Is there a way (option, function) to better handle large files?
 
 Seemingly SAS can deal with it with no problem, but I just persuaded my
 professor transfering to R, so it is quite disappointing.
 
 Please help, thank you.
 
   [[alternative HTML version deleted]]
 
 __
 R-help@stat.math.ethz.ch mailing list
 https://stat.ethz.ch/mailman/listinfo/r-help
 PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
 and provide commented, minimal, self-contained, reproducible code.

__
R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.