Mulholland, Tom <[EMAIL PROTECTED]> wrote: > > I would suggest that you make a more thorough search of the > R-Archives. > (http://finzi.psych.upenn.edu/search.html) If you do you will find > this > discussion has been had several times and that the type of machine > you > are running will have an impact upon what you can do. My feeling is > that > you are going have to knuckle down with the documentation and > understand > how R works and then when you have specific issues that show you have > read all the appropriate documentation, you might try another message > to > the list. > > Ciao, Tom
Another approach is to not try to bring all your data into R at once - it is unlikely that you actually need every column of every row in your dataset to undertake any particular analysis. The trick is to bring into R only those rows and columns which you need at a particular moment, and then discard them. The best way to do this is to manage your data in an SQl database such as MySQL or PostgreSQL, and then use one of the R database interfaces to issue queries against this database and to surface the query results as a data frame. Remeber to compose your queries in such as way as to only retreive the rows and columns you really need at any particular moment, and don't forget to delete these data frames as soon as you have finished with them (or at least, as soon as you need more space in your R session). There is also an (experimental I think) package which allows lazy or virtual loading of database queries into data frames, so that the query results are paged into memory as they are needed. But I doubt you will need that. Tim C > > _________________________________________________ > > Tom Mulholland > Senior Policy Officer > WA Country Health Service > Tel: (08) 9222 4062 > > The contents of this e-mail transmission are confidential and may be > protected by professional privilege. The contents are intended only > for > the named recipients of this e-mail. If you are not the intended > recipient, you are hereby notified that any use, reproduction, > disclosure or distribution of the information contained in this > e-mail > is prohibited. Please notify the sender immediately. > > > -----Original Message----- > From: Edward McNeil [mailto:[EMAIL PROTECTED] > Sent: Tuesday, 2 December 2003 8:45 AM > To: [EMAIL PROTECTED] > Subject: [R] R and Memory > > > Dear all, > This is my first post. > We have started to use R here and have also started teaching it to > our > PhD students. Our unit will be the HQ for developing R throughout > Thailand. > > I would like some help with a problem we are having. We have one > sample > of data that is quite large in fact - over 2 million records (ok ok > it's > more like a population!). The data is stored in SPSS. The file is > over > 350Mb but SPSS happily stores this much data. Now when I try to read > it > into R it grunts and groans for a few seconds and then reports that > there is not enough memory (the computer has 250MB RAM). I have tried > setting the memory in the command line (--max-vsize and > --max-mem-size) > but all to no avail. > > Any help would be muchly appreciated! > > Edward McNeil (son of Don) > Epidemiology Unit > Faculty of Medicine > Prince of Songkhla University > Hat Yai 90110 > THAILAND > > ______________________________________________ > [EMAIL PROTECTED] mailing list > https://www.stat.math.ethz.ch/mailman/listinfo/r-help > > ______________________________________________ > [EMAIL PROTECTED] mailing list > https://www.stat.math.ethz.ch/mailman/listinfo/r-help ______________________________________________ [EMAIL PROTECTED] mailing list https://www.stat.math.ethz.ch/mailman/listinfo/r-help
