Re: [R] It is possible to Expand RAM with HD space?
Hi, (closely related to the subject) First, I'm not an expert in programming, so the terminology I use can be incorrect or vague. I wonder if it's a good development to always defend from out of memory error by buying more RAM because the size of analyseable data also increases at same time. In addition (I see) a unwise development of operating systems (at least MS) sets higher and higher requirements for the machine (which of course is not the R's fault) For the future, would it be possible to build a data/object file system parallel to R workspace in which you typically could avoid loading all the data into use memory. So to have a very scalable solution to the painful memory problem. I am using R in a strong relationship with a Finnish statistical system Survo (www.survo.fi) In the C-programmed Survo the data or matrix files are usually never wholly loaded into memory. Minimally you can load just one element from the data file (via C-function) into memory (e.g. for calculating some statistics). Of course in that way (since the data is not in fast memory) the operations are slower compared to operating with R objects. But the other (and important) side of the coin is that the size of free hard disk space is practically the only limit in Survo (of course some methods (for computational reasons) require a loading of bigger part of the data into fast memory). I also see that a marriage with some database system or data warehouse can sometimes cause a creative analysis work a bit clumsy. Example: How I have used R within Survo variable transformations. In this way you don't need to load all the data into R workspace (just the vectors of interest). A small example data in Survo editor where all the operations are done: DATA _TMP1 X Y 1 A 2 A 3 B 4 B 5 B FILE COPY _TMP1 TO TMP1 / copy to Survo data file FILE STATUS TMP1 / Structure of data Copy of data matrix _TMP1 FIELDS: (active) 1 NA_ 1 X(#) 2 SA_ 1 Y END Survo data file TMP1: record=22 bytes, M1=6 L=64 M=2 N=5 Variable transformations: CASE 1: Survo's own VAR operation VAR X2=log(X) TO TMP1 / Activation in the editor FILE LOAD TMP1 DATA TMP1* X Y X2 1 A 0.000 2 A 0.693 3 B 1.099 4 B 1.386 5 B 1.609 CASE 2: Identical VAR operation via R function (using a sucro tehcnique (~macro) in Survo) /R_VAR X2b=c(log(TMP1$X)) TO TMP1 / for your comments etc. FILE LOAD TMP1 / see if the equal results ;) DATA TMP1* X Y X2 X2b 1 A 0.000 0.000 2 A 0.693 0.693 3 B 1.099 1.099 4 B 1.386 1.386 5 B 1.609 1.609 where /R_VAR is the sucro (I haven't listed the code here) carrying out the visit in R workspace (log()) and then returning the output vector into Survo data file TMP1. Perhaps any R script that results a vector of the data row dim can be used as an argument for this simple sucro function. I'm very excited about data processing with R. I see my approach just to union the strengths of these two programs. Sometimes with larger data I do the pre-processing mainly with Survo and then the final analyses (e.g. lmer()) in R perhaps in connection with a database (though Survo covers a wide set of statistical operations, the matrix interpreter is great and it's easy do your C-modules for Survo). Perhaps this all is also possible with SAS or SPSS (I my opinion SPSS is inconvienent and SAS I have never used but heard that it's at least overpriced ;-) Best regards, Petri -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Prof Brian Ripley Sent: 17. huhtikuuta 2007 11:19 To: Uwe Ligges Cc: r-help@stat.math.ethz.ch Subject: Re: [R] It is possible to Expand RAM with HD space? On Tue, 17 Apr 2007, Uwe Ligges wrote: Jorge Cornejo-Donoso wrote: I'm using R in a winXp machine. I have problems with the size of my database (xyz), anyone know if is possible to use some kind of swap memory, in order to expand my real RAM and use HD space for the processes? This is done automatically by your operating system. You do not need to care about it. Except perhaps that on his OS it is limited to 2GB, which can be increased to 3GB in some circumstances (see the rw-FAQ Q2.9). And anyone using less than 2GB RAM at current prices (except perhaps on a laptop) and is running out of RAM needs to buy some more: *far* cheaper than buying SAS. Uwe Ligges I'm trying to make a variogram, and I don't want to sacrifice the precision of the data, e.g. using mean or median by defined pixels or something like that (now I have to do that, or use other software like SAS). Thanks in advance! __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained,
Re: [R] It is possible to Expand RAM with HD space?
Jorge Cornejo-Donoso wrote: I'm using R in a winXp machine. I have problems with the size of my database (xyz), anyone know if is possible to use some kind of swap memory, in order to expand my real RAM and use HD space for the processes? This is done automatically by your operating system. You do not need to care about it. Uwe Ligges I'm trying to make a variogram, and I don't want to sacrifice the precision of the data, e.g. using mean or median by defined pixels or something like that (now I have to do that, or use other software like SAS). Thanks in advance! __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] It is possible to Expand RAM with HD space?
On Tue, 17 Apr 2007, Uwe Ligges wrote: Jorge Cornejo-Donoso wrote: I'm using R in a winXp machine. I have problems with the size of my database (xyz), anyone know if is possible to use some kind of swap memory, in order to expand my real RAM and use HD space for the processes? This is done automatically by your operating system. You do not need to care about it. Except perhaps that on his OS it is limited to 2GB, which can be increased to 3GB in some circumstances (see the rw-FAQ Q2.9). And anyone using less than 2GB RAM at current prices (except perhaps on a laptop) and is running out of RAM needs to buy some more: *far* cheaper than buying SAS. Uwe Ligges I'm trying to make a variogram, and I don't want to sacrifice the precision of the data, e.g. using mean or median by defined pixels or something like that (now I have to do that, or use other software like SAS). Thanks in advance! __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- Brian D. Ripley, [EMAIL PROTECTED] Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/ University of Oxford, Tel: +44 1865 272861 (self) 1 South Parks Road, +44 1865 272866 (PA) Oxford OX1 3TG, UKFax: +44 1865 272595 __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] It is possible to Expand RAM with HD space?
I'm using R in a winXp machine. I have problems with the size of my database (xyz), anyone know if is possible to use some kind of swap memory, in order to expand my real RAM and use HD space for the processes? I'm trying to make a variogram, and I don't want to sacrifice the precision of the data, e.g. using mean or median by defined pixels or something like that (now I have to do that, or use other software like SAS). Thanks in advance! __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
