Re: [R] Error in vector(double, length) : vector size specified is too large....VLDs
It took about 2 hours on the 64 bit windows platform. Now I just need to find my notes from ST733 and remember how to use GSTAT to estimate the parameters http://www4.ncsu.edu/~tpcolson/variog.jpg Thomas Colson North Carolina State University Department of Forestry and Environmental Resources (919) 673 8023 [EMAIL PROTECTED] Calendar: www4.ncsu.edu/~tpcolson -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Edzer J. Pebesma Sent: Friday, September 16, 2005 5:27 AM To: r-help@stat.math.ethz.ch Subject: [R] Error in vector(double, length) : vector size specified is too largeVLDs Tom, please try to use the variogram function in package gstat; it doesn't (try to) store all pairwise differences, but rather accumulates them for distance intervals. It will take a while to do this, and there is a chance that you overflow the counter that keeps the number of point pairs for each interval: 304000^2 2^32; it is stored as a C long, so may work on a 64 bit architecture. Otherwise, I'd suggest to sample your data set. I'd be interested to hear whether you succeed (or not). -- Edzer __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
[R] Error in vector(double, length) : vector size specified is too large....VLDs
I have what R seems to consider a very large dataset, a 12MB text file of lat,long,and height values, 130,000 rows to be exact. Here's what I get: Thomas Colson North Carolina State University Department of Forestry and Environmental Resources (919) 673 8023 [EMAIL PROTECTED] Calendar: www4.ncsu.edu/~tpcolson __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
[R] Error in vector(double, length) : vector size specified is too large....VLDs
I have what R seems to consider a very large dataset, a 12MB text file of lat,long,and height values, 130,000 rows to be exact. Here's what I get: data1 - data.frame(read.table(BE3720078500WC20020828.txt,sep=,, header=T)) raw.data - as.geodata(data1) variog.1.b - variog(raw.data) variog: computing omnidirectional variogram Error in vector(double, length) : vector size specified is too large round(memory.limit()/1048576.0, 2) [1] 4000 The Vector size specified is too large seems to be a common error, but I haven't seen any workarounds posted...and the help.archive web site seems to be down. I can plot the dataset, do some elementary stats on it...no variogram though. Any ideas on how to compute variograms on datasets with 100 to 300k points? Thanks Thomas Colson North Carolina State University Department of Forestry and Environmental Resources (919) 673 8023 [EMAIL PROTECTED] Calendar: www4.ncsu.edu/~tpcolson __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Re: [R] Error in vector(double, length) : vector size specified is too large....VLDs
At 4 GB, I'm at the 32bit windows limit. Thomas Colson North Carolina State University Department of Forestry and Environmental Resources (919) 673 8023 [EMAIL PROTECTED] Calendar: www4.ncsu.edu/~tpcolson -Original Message- From: Berton Gunter [mailto:[EMAIL PROTECTED] Sent: Thursday, September 15, 2005 2:34 PM To: 'Tom Colson' Subject: RE: [R] Error in vector(double,length) : vector size specified is too largeVLDs Any ideas on how to compute variograms on datasets with 100 to 300k points? Thanks Get more memory? ... it's cheap! :-) -- Bert Gunter Genentech __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Re: [R] Error in vector(double, length) : vector size specified is too large....VLDs
rm(data1) variog.1.b - variog(raw.data) variog: computing omnidirectional variogram Error in vector(double, length) : vector size specified is too large Turns out I was wrong re: # of rows...it's 304,000 Same problem. Version is 2.1.1, hardware is Dual Xeon 3.6 4 GB RAM, XP Pro 64 Bit. Can reproduce the problem with 64Bit R 2.1.1 running on Fedora 4, same hardware. Thomas Colson North Carolina State University Department of Forestry and Environmental Resources (919) 673 8023 [EMAIL PROTECTED] Calendar: www4.ncsu.edu/~tpcolson -Original Message- From: Douglas Grove [mailto:[EMAIL PROTECTED] Sent: Thursday, September 15, 2005 2:23 PM To: Tom Colson Subject: Re: [R] Error in vector(double, length) : vector size specified is too largeVLDs Well you could start by removing large objects that you aren't using (e.g. 'data1') and seeing if that helps. There may be other suggestions but you haven't told us what platform you're working on, as the posting guide requests: PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html Doug On Thu, 15 Sep 2005, Tom Colson wrote: I have what R seems to consider a very large dataset, a 12MB text file of lat,long,and height values, 130,000 rows to be exact. Here's what I get: data1 - data.frame(read.table(BE3720078500WC20020828.txt,sep=,, header=T)) raw.data - as.geodata(data1) variog.1.b - variog(raw.data) variog: computing omnidirectional variogram Error in vector(double, length) : vector size specified is too large round(memory.limit()/1048576.0, 2) [1] 4000 The Vector size specified is too large seems to be a common error, but I haven't seen any workarounds posted...and the help.archive web site seems to be down. I can plot the dataset, do some elementary stats on it...no variogram though. Any ideas on how to compute variograms on datasets with 100 to 300k points? Thanks Thomas Colson North Carolina State University Department of Forestry and Environmental Resources (919) 673 8023 [EMAIL PROTECTED] Calendar: www4.ncsu.edu/~tpcolson __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Re: [R] Error in vector(double, length) : vector size specified is too large....VLDs
Yes, using geoR. I can interpolate the DEM quite easily in Grass (v.surf.rst, kriging) and block kriging in ArcInfo. What we need, though, is to be able to estimate or even nail down the variogram for these data sets. Where am I going with this? I'm guessing that variables such as slope, ruggedness, etc.. are going to produce different sill, range, and nugget values, which I can then use to fine tune the interpolation process, rather than using the same spline or kriging parameters on say, a whole state boundary worth of Lidar data. And yes, I can estimate the variogram in ArcInfo (limited to 1 points) and can also import the DEM from grass into R using spgrassbut the point is to analyze the point data BEFORE I make the DEM. So I'm guessing the geoR isn't ever going to handle this size data, and I need to be using gstat? (As I write this, gstat(variogram) is plugging away for last 10 minute with no errors.) Thanks for quick replies Thomas Colson North Carolina State University Department of Forestry and Environmental Resources (919) 673 8023 [EMAIL PROTECTED] Calendar: www4.ncsu.edu/~tpcolson -Original Message- From: Roger Bivand [mailto:[EMAIL PROTECTED] Sent: Thursday, September 15, 2005 3:28 PM To: Peter Dalgaard Cc: Tom Colson; r-help@stat.math.ethz.ch Subject: Re: [R] Error in vector(double, length) : vector size specified is too largeVLDs On 15 Sep 2005, Peter Dalgaard wrote: Tom Colson [EMAIL PROTECTED] writes: rm(data1) variog.1.b - variog(raw.data) variog: computing omnidirectional variogram Error in vector(double, length) : vector size specified is too large Turns out I was wrong re: # of rows...it's 304,000 Same problem. Version is 2.1.1, hardware is Dual Xeon 3.6 4 GB RAM, XP Pro 64 Bit. Can reproduce the problem with 64Bit R 2.1.1 running on Fedora 4, same hardware. Variograms involve the differences between all pairs of points which can become a rather large number of values. 304000*303999/2 in your case, about 344GB by my reckoning. And the distances between them makes for a similar quantity. Now, some algorithms may be smarter than to keep all values in memory, but you haven't even told us where you got the variog() from. It doesn't seem to be in the standard packages, although we do have variogram() and Variogram() in spatial and nlme. Right, this is from geoR, which uses full matrices. I think both fields and gstat can work with larger data sets. Whether model-based geostatistics is what you need for interpolating a digital elevation model is another question. -- Roger Bivand Economic Geography Section, Department of Economics, Norwegian School of Economics and Business Administration, Helleveien 30, N-5045 Bergen, Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43 e-mail: [EMAIL PROTECTED] __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
[R] BLAS or ATLAS?
I'm trying to tweak multi-threading on a multiprocessor box(Xeon). Reading appendix a.2.2 in the install and admin menu, I see that I can use the internal BLAS lib and/or a multi-thread version of ATLAS (Where to get?) Question: Intel claims their libraries are faster than ATLAS. Has anyone compiled R against the Intel BLAS? If so...how did you do it? http://developer.intel.com/software/products/mkl/features/lin_alg.htm And I'm in the market for a pre-built Atlas dual processor lib for Xeon. Know where I can find one? Thanks. -- Tom Colson Center for Earth Observation North Carolina State University Raleigh, NC 27695 (919) 515 3434 (919) 673 8023 [EMAIL PROTECTED] Online Calendar: http://www4.ncsu.edu/~tpcolson __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Re: [R] BLAS or ATLAS?
Fedora Core 3 I installed the Intel MLk, and tried : --with-blas=-lmkl -lguide -lpthread and got : External libraries:readline, BLAS(generic) thus I'm assuming I'm missing something when telling config where to look for the recently installed Intel BLAS? thanks for all replies thus far. On Wed, 2005-02-23 at 19:25 +, Prof Brian Ripley wrote: What OS is this for? Several OSes (Windows, Linux, Solaris, FreeBSD ...) run on Xeons (and there are several sorts of Xeons, some even EM64T AMD-clones). On Wed, 23 Feb 2005, Tom Colson wrote: I'm trying to tweak multi-threading on a multiprocessor box(Xeon). Reading appendix a.2.2 in the install and admin menu, I see that I can use the internal BLAS lib and/or a multi-thread version of ATLAS (Where to get?) The last _is_ described in Appendix A.2.2, together with several other possibilities. Question: Intel claims their libraries are faster than ATLAS. Has anyone compiled R against the Intel BLAS? If so...how did you do it? http://developer.intel.com/software/products/mkl/features/lin_alg.htm That _is_ described in Appendix A.2.2. And I'm in the market for a pre-built Atlas dual processor lib for Xeon. Know where I can find one? __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Re: [R] BLAS or ATLAS?
In makeconf I get BLAS_LIBS = -lblas I believe that actually means configure was able to use the BLAS you specified (MKL in this case). You can scan through config.log to make sure. in config.logcan't seem to find any reference to the Intel BLAS. Thanks for all the replies. On Wed, 2005-02-23 at 13:45 -0600, Douglas Bates wrote: Tom Colson wrote: Fedora Core 3 I installed the Intel MLk, and tried : --with-blas=-lmkl -lguide -lpthread and got : External libraries:readline, BLAS(generic) thus I'm assuming I'm missing something when telling config where to look for the recently installed Intel BLAS? That's what you would expect to see in the summary section from configure. Look back in the configure output to see exactly what it reported for the blas. You may also check the Makeconf file that was generated to see what it defines for BLAS_LIBS __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
[R] R GUI for Gnome?
Just installed R 64 Bit precompiled binaries on Fedora Core 3 (Kernel 2.6.10)http://cran.stat.ucla.edu/bin/linux/redhat/fc3/x86_64/R-2.0.1-0.f dr.1.x86_64.rpm And R seem to start right up. ProblemI a windows slave...and miss the handy R gui that come with the windows version (for doing repetitive tasks like loading packages and changing directories). If I try R -g XLL...it still starts in the terminal mode. R --gui-gnome returns GNOME GUI not available in this verions. A read of the manual reveals that I must compile GNOME Support with ./configure --with-gnome. But I'm at a loss as to how to configure/compile R for 64 Bit using the source. Thanks. __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
RE: [R] Precompiled x86_64 Binaries?
Yes, I meant to say Suse. I had tried Fedorabut I'm one of those who has to have bleeding edge hardware...for which there are no working Fedora Drivers. Hence the switch to Suse...so I can get video..so I can get a 64 Bit R running. All of this is just a test. If we can get 64Bit R compiled and running here...then we'll go to the Blade center with a procedure log on install and repeat it on the Grid Computer. Thanks. -Original Message- From: Prof Brian Ripley [mailto:[EMAIL PROTECTED] Sent: Friday, February 18, 2005 10:15 AM To: Thomas Colson Cc: r-help@stat.math.ethz.ch Subject: Re: [R] Precompiled x86_64 Binaries? Yes, under redhat/fc3. Did you forget to mention SuSE? On Fri, 18 Feb 2005, Thomas Colson wrote: Can't seem to find a x86_64 RPM for R on any of the DL mirrors. Does one exist? From: Peter Dalgaard p.dalgaard_at_biostat.ku.dk Date: Fri 22 Oct 2004 - 23:00:41 EST Prof Brian Ripley [EMAIL PROTECTED] writes: R does not run under (beta) 64-bit Windows on x86_64 and we have no plans to port it there, but it does run under 32-bit Windows on that processor. I am sure it will run on several 64-bit operating systems on Apple Macintosh's so-called G5 hardware -- I believe we have seen a report that includes an in-progress version of MacOS 10.4 -- and also several 32-bit ones. Re. Windows, the issue is that we're using the mingw32 toolkit. If there is ever a mingw64, I'm sure we'd attempt a port almost immediately. If you want a prebuilt version you are out of luck except for Debian Linux on Alpha or ia64, from a quick glance. Actually, I've rolled up an x86_64 RPM for SuSE 9.1 which should be en route to CRAN. Detlef Steuer is awaiting delivery of a 64bit machine and plan to maintain the SuSE version in the longer run. It's sitting in http://www.biostat.ku.dk/~pd/R-suse9.1-x86_64 for now. In general, it seems to be quite trivial to take a SPEC or SRPM for the 32 bit version of a distribution and rebuild an RPM for 64 bits. However, you might prefer to install from the source tarball anyway if you want to take advantage of high-performance libraries (some of which can not be redistributed). -- Brian D. Ripley, [EMAIL PROTECTED] Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/ University of Oxford, Tel: +44 1865 272861 (self) 1 South Parks Road, +44 1865 272866 (PA) Oxford OX1 3TG, UKFax: +44 1865 272595 __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
