Thanks a lot Jim. Forwarded to Aaron (who rised the question) and the R-help list...
Patrick jim holtman a écrit : > The size matrix you are allocating (18227 x 18227) would require 2.6GB > of memory and that is what the error message is saying that you only > have 0.5GB (512MB) available. You also need about 3-4x the largest > object so that you can do any calculations on it due to extra copies > that might be made of the data. > > Now your 2000 x 2000 will require about 32MB of memory that that is > why it is comfortable in running on your system with 512MB. > > If you are running on a Windows machine, you may want to use the > --max-memory command line parameter to set the memory size. I have > 1GB and set the max to 800MB. You may also want to remove any extra > objects and do 'gc()' before working with a large object to try and > get memory cleared up. > > > On 1/5/06, *Patrick Giraudoux* <[EMAIL PROTECTED] > <mailto:[EMAIL PROTECTED]>> wrote: > > Dear Aaron, > > I am really a tool user and not a tool maker (actually an ecologist > doing some biostatistics)... so, I take the liberty of sending a > copy of > this e-mail to the r-help list where capable computer persons and true > statisticians may provide more relevant information and also to Paulo > Ribeiro and Peter Diggle, the authors of geoR.. > > I really feel that your huge matrix cannot be handled in R that easy, > and I get the same kind of error as you: > > > m=matrix(1,ncol=18227, nrow=18227) > Error: cannot allocate vector of size 2595496 Kb > In addition: Warning messages: > 1: Reached total allocation of 511Mb: see help( memory.size) > 2: Reached total allocation of 511Mb: see help(memory.size) > > However, if you want to compute a distance matrix, have a look to the > function: > > ?dist > > and try it... You will not have to create the distance matrix > yourself > from the coordinates file (but you may meet meory problem anyway). > Still > more straigthfully, if you intend to use interpolation methods such as > kriging, you don't need to manage the distance matrix building by > yourself. See: > > library(geoR) > library(help=geoR) > ?variog > ?variofit > ?likfit? > > etc... > > If you want further use geostatics (eg via geoR or gstat), you will > anyway have to manage with memory limits (not due to R). On my > computer > (portable HP compaq nx7000) I can hardly manage with more than 2000 > observations using geoR (far from your 18227 observations). I had to > krige on a dataset with 9000 observations recently and has been led to > subsample randomly 2000 values. You can however try and increase the > memory allocated to R on your computer. The size limit is hardware > dependent, eg: > > ?memory.size > ?memory.limit > / memory.limit(size=500000000) / > > Another way may be to perform local kriging (eg kriging within the > dataset on a fixed radius), but to my knowldege this cannot be > done with > geoR (unfortunately). The library gstat offers this option, but > has much > more limited possibilities than geoR considering other issues > (variogram > analysis, etc...). > > library(gstat) > ?krige > > see argument 'maxdist' > > Hope this can help, > > Kind regards > > > Patrick Giraudoux > -- > > Department of Environmental Biology > EA3184 usc INRA > University of Franche-Comte > 25030 Besancon Cedex > (France) > > tel. +33 381 665 745 > fax +33 381 665 797 > http://lbe.univ-fcomte.fr > > > Aaron Swoboda a écrit : > > > Dear Sir: > > > > > > I ran across your post to the R-help archive from February 9it is > > attached below since it was nearly two years ago!). I am > beginning to > > learn R and am interested in analyzing some of my data in a spatial > > context. I am having a problem that seems similar to the problem you > > encountered, trying to work with a large matrix in Windows XP). > I am > > wondering if you could help steer me in the direction that > helped you > > solve your problem. I am trying to construct a distance matrix > > containing the all of the distances between my 18,000 observations. > > Trying to make a matrix that large gets an error message... > > > > m=matrix(1,ncol=18227, nrow=18227) > > > > Error: cannot allocate vector of size 2595496 Kb > > In addition: Warning messages: > > 1: Reached total allocation of 1023Mb: see help(memory.size) > > > > Thank you for any help you may be able to send my way. > > > > ~Aaron Swoboda > > > > Below is your posting to the R-help list... > > > > > > Dear all, > > > > I a read with great interest the e-mails related to Arnav Sheth > about > > memory limitation when computing a distance matrix. I suspect > > that I will also meet some memory limitation using GeoR. I am > > currently running GeoR on a geodata object including 2686 > geographical > > coordinates. > > > > krige.conv() can handle it (it takes 10-15 mn of computing) but > > requests an increased memory. > > > > /> memory.limit(size=500000000) / > > > > When the computing is completed, the computer speed is considerably > > slowed down for any application. It is thus most necessary to > > shut it down and restart. I will probably have to handle a set of > > 5000-6000 coordinates in once in the next few months. I wonder if > > it will go through it on my plateform (Windows XP and compaq > nx7000). > > If not, will the limitation due to R or to Windows? Does an > > alternate solution exist? > > > > Thanks for any hint, > > > > Patrick Giraudoux > > > >-- > > Dr. Aaron M. Swoboda > > 3208 Posvar Hall > > Graduate School of Public and International Affairs > > University of Pittsburgh > > Pittsburgh, PA 15260 > > > > e-mail: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]> > > Office: 412-648-7604 > > Fax: 412-648-2605 > > > > > > [[alternative HTML version deleted]] > > > > ______________________________________________ > [email protected] <mailto:[email protected]> mailing > list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide! > http://www.R-project.org/posting-guide.html > > > > > -- > Jim Holtman > Cincinnati, OH > +1 513 247 0281 > > What the problem you are trying to solve? [[alternative HTML version deleted]]
______________________________________________ [email protected] mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
