Re: [R] Using very large matrix
Thanks a lot! Unfortunately, the R package I have to sue for my research was only released on 32 bit R on 32 bit MS Windows and only closed source I normally use 64 bit R on 64 bit Linux :) I tried to use the bigmemory in cran with 32 bit windows, but I had some serious problems. Best, On Thursday 26 February 2009 15:43:11 Jay Emerson wrote: Corrado, Package bigmemory has undergone a major re-engineering and will be available soon (available now in Beta version upon request). The version currently on CRAN is probably of limited use unless you're in Linux. bigmemory may be useful to you for data management, at the very least, where x - filebacked.big.matrix(8, 8, init=n, type=double) would accomplish what you want using filebacking (disk space) to hold the object. But even this requires 64-bit R (Linux or Mac, or perhaps a Beta version of Windows 64-bit R that REvolution Computing is working on). Subsequent operations (e.g. extraction of a small portion for analysis) are then easy enough: y - x[1,] would give you the first row of x as an object y in R. Note that x is not itself an R matrix, and most existing R analytics can't work on x directly (and would max out the RAM if they tried, anyway). Feel free to email me for more information (and this invitation applies to anyone who is interested in this). Cheers, Jay #Dear friends, # #I have to use a very large matrix. Something of the sort of #matrix(8,8,n) where n is something numeric of the sort 0.xx # #I have not found a way of doing it. I keep getting the error # #Error in matrix(nrow = 8, ncol = 8, 0.2) : too many elements specified # #Any suggestions? I have searched the mailing list, but to no avail. # #Best, #-- #Corrado Topi # #Global Climate Change Biodiversity Indicators #Area 18,Department of Biology #University of York, York, YO10 5YW, UK #Phone: + 44 (0) 1904 328645, E-mail: ct...@york.ac.uk -- Corrado Topi Global Climate Change Biodiversity Indicators Area 18,Department of Biology University of York, York, YO10 5YW, UK Phone: + 44 (0) 1904 328645, E-mail: ct...@york.ac.uk __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Using very large matrix
I'm very interested in the bigmemory package for windows 32-bit environments. Who do I need to contact to request the Beta version? Thanks Steve Steve Friedman Ph. D. Spatial Statistical Analyst Everglades and Dry Tortugas National Park 950 N Krome Ave (3rd Floor) Homestead, Florida 33034 steve_fried...@nps.gov Office (305) 224 - 4282 Fax (305) 224 - 4147 Corrado ct...@york.ac.uk To Sent by: john.emer...@yale.edu, Tony Breyal r-help-boun...@r- tony.bre...@googlemail.com project.orgcc r-help@r-project.org Subject 03/02/2009 10:46 Re: [R] Using very large matrix AM GMT Thanks a lot! Unfortunately, the R package I have to sue for my research was only released on 32 bit R on 32 bit MS Windows and only closed source I normally use 64 bit R on 64 bit Linux :) I tried to use the bigmemory in cran with 32 bit windows, but I had some serious problems. Best, On Thursday 26 February 2009 15:43:11 Jay Emerson wrote: Corrado, Package bigmemory has undergone a major re-engineering and will be available soon (available now in Beta version upon request). The version currently on CRAN is probably of limited use unless you're in Linux. bigmemory may be useful to you for data management, at the very least, where x - filebacked.big.matrix(8, 8, init=n, type=double) would accomplish what you want using filebacking (disk space) to hold the object. But even this requires 64-bit R (Linux or Mac, or perhaps a Beta version of Windows 64-bit R that REvolution Computing is working on). Subsequent operations (e.g. extraction of a small portion for analysis) are then easy enough: y - x[1,] would give you the first row of x as an object y in R. Note that x is not itself an R matrix, and most existing R analytics can't work on x directly (and would max out the RAM if they tried, anyway). Feel free to email me for more information (and this invitation applies to anyone who is interested in this). Cheers, Jay #Dear friends, # #I have to use a very large matrix. Something of the sort of #matrix(8,8,n) where n is something numeric of the sort 0.xx # #I have not found a way of doing it. I keep getting the error # #Error in matrix(nrow = 8, ncol = 8, 0.2) : too many elements specified # #Any suggestions? I have searched the mailing list, but to no avail. # #Best, #-- #Corrado Topi # #Global Climate Change Biodiversity Indicators #Area 18,Department of Biology #University of York, York, YO10 5YW, UK #Phone: + 44 (0) 1904 328645, E-mail: ct...@york.ac.uk -- Corrado Topi Global Climate Change Biodiversity Indicators Area 18,Department of Biology University of York, York, YO10 5YW, UK Phone: + 44 (0) 1904 328645, E-mail: ct...@york.ac.uk __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Using very large matrix
Steve et.al., The old version is still on CRAN, but I strongly encourage anyone interested to email me directly and I'll make the new version available. In fact, I wouldn't mind just pulling the old version off of CRAN, but of course that's not a great idea. !-) Jay On Mon, Mar 2, 2009 at 8:47 AM, steve_fried...@nps.gov wrote: I'm very interested in the bigmemory package for windows 32-bit environments. Who do I need to contact to request the Beta version? Thanks Steve Steve Friedman Ph. D. Spatial Statistical Analyst Everglades and Dry Tortugas National Park 950 N Krome Ave (3rd Floor) Homestead, Florida 33034 steve_fried...@nps.gov Office (305) 224 - 4282 Fax (305) 224 - 4147 Corrado ct...@york.ac.uk To Sent by: john.emer...@yale.edu, Tony Breyal r-help-boun...@r- tony.bre...@googlemail.com project.org cc r-help@r-project.org Subject 03/02/2009 10:46 Re: [R] Using very large matrix AM GMT Thanks a lot! Unfortunately, the R package I have to sue for my research was only released on 32 bit R on 32 bit MS Windows and only closed source I normally use 64 bit R on 64 bit Linux :) I tried to use the bigmemory in cran with 32 bit windows, but I had some serious problems. Best, On Thursday 26 February 2009 15:43:11 Jay Emerson wrote: Corrado, Package bigmemory has undergone a major re-engineering and will be available soon (available now in Beta version upon request). The version currently on CRAN is probably of limited use unless you're in Linux. bigmemory may be useful to you for data management, at the very least, where x - filebacked.big.matrix(8, 8, init=n, type=double) would accomplish what you want using filebacking (disk space) to hold the object. But even this requires 64-bit R (Linux or Mac, or perhaps a Beta version of Windows 64-bit R that REvolution Computing is working on). Subsequent operations (e.g. extraction of a small portion for analysis) are then easy enough: y - x[1,] would give you the first row of x as an object y in R. Note that x is not itself an R matrix, and most existing R analytics can't work on x directly (and would max out the RAM if they tried, anyway). Feel free to email me for more information (and this invitation applies to anyone who is interested in this). Cheers, Jay #Dear friends, # #I have to use a very large matrix. Something of the sort of #matrix(8,8,n) where n is something numeric of the sort 0.xx # #I have not found a way of doing it. I keep getting the error # #Error in matrix(nrow = 8, ncol = 8, 0.2) : too many elements specified # #Any suggestions? I have searched the mailing list, but to no avail. # #Best, #-- #Corrado Topi # #Global Climate Change Biodiversity Indicators #Area 18,Department of Biology #University of York, York, YO10 5YW, UK #Phone: + 44 (0) 1904 328645, E-mail: ct...@york.ac.uk -- Corrado Topi Global Climate Change Biodiversity Indicators Area 18,Department of Biology University of York, York, YO10 5YW, UK Phone: + 44 (0) 1904 328645, E-mail: ct...@york.ac.uk __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- John W. Emerson (Jay) Assistant Professor of Statistics Department of Statistics Yale University http://www.stat.yale.edu/~jay __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Using very large matrix
Corrado, Package bigmemory has undergone a major re-engineering and will be available soon (available now in Beta version upon request). The version currently on CRAN is probably of limited use unless you're in Linux. bigmemory may be useful to you for data management, at the very least, where x - filebacked.big.matrix(8, 8, init=n, type=double) would accomplish what you want using filebacking (disk space) to hold the object. But even this requires 64-bit R (Linux or Mac, or perhaps a Beta version of Windows 64-bit R that REvolution Computing is working on). Subsequent operations (e.g. extraction of a small portion for analysis) are then easy enough: y - x[1,] would give you the first row of x as an object y in R. Note that x is not itself an R matrix, and most existing R analytics can't work on x directly (and would max out the RAM if they tried, anyway). Feel free to email me for more information (and this invitation applies to anyone who is interested in this). Cheers, Jay #Dear friends, # #I have to use a very large matrix. Something of the sort of #matrix(8,8,n) where n is something numeric of the sort 0.xx # #I have not found a way of doing it. I keep getting the error # #Error in matrix(nrow = 8, ncol = 8, 0.2) : too many elements specified # #Any suggestions? I have searched the mailing list, but to no avail. # #Best, #-- #Corrado Topi # #Global Climate Change Biodiversity Indicators #Area 18,Department of Biology #University of York, York, YO10 5YW, UK #Phone: + 44 (0) 1904 328645, E-mail: ct...@york.ac.uk -- John W. Emerson (Jay) Assistant Professor of Statistics Department of Statistics Yale University http://www.stat.yale.edu/~jay __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Using very large matrix
I have to use a very large matrix. Something of the sort of matrix(8,8,n) where n is something numeric of the sort 0.xx I have not found a way of doing it. I keep getting the error Error in matrix(nrow = 8, ncol = 8, 0.2) : too many elements specified Any suggestions? I have searched the mailing list, but to no avail. A 8x8 matrix has 6.4 billion cells. If you assume 4 byte (32bit) for a double precision floating point number that's an impressive 25.6 Gb. Certainly does not fit into RAM on my machine. cu Philipp -- Dr. Philipp Pagel Lehrstuhl für Genomorientierte Bioinformatik Technische Universität München Wissenschaftszentrum Weihenstephan 85350 Freising, Germany http://mips.gsf.de/staff/pagel __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Using very large matrix
Philipp Pagel wrote: I have to use a very large matrix. Something of the sort of matrix(8,8,n) where n is something numeric of the sort 0.xx I have not found a way of doing it. I keep getting the error Error in matrix(nrow = 8, ncol = 8, 0.2) : too many elements specified Any suggestions? I have searched the mailing list, but to no avail. A 8x8 matrix has 6.4 billion cells. If you assume 4 byte (32bit) for a double precision floating point number that's an impressive 25.6 Gb. Certainly does not fit into RAM on my machine. Doubles are 8 bytes, and R can't handle vectors that large, even if you have the memory. (Indices are limited to 32 bit values even in 64 bit R.) Duncan Murdoch __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Using very large matrix
On Wed, Feb 25, 2009 at 06:37:53AM -0500, Duncan Murdoch wrote: Philipp Pagel wrote: A 8x8 matrix has 6.4 billion cells. If you assume 4 byte (32bit) for a double precision floating point number that's an impressive 25.6 Gb. Certainly does not fit into RAM on my machine. Doubles are 8 bytes, Oops - of course. Even worse... cu Philipp -- Dr. Philipp Pagel Lehrstuhl für Genomorientierte Bioinformatik Technische Universität München Wissenschaftszentrum Weihenstephan 85350 Freising, Germany http://mips.gsf.de/staff/pagel __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Using very large matrix
looks like you've run out of memory mate, because that sure is a big matrix, you'd probably need 64 bit OS/CPU/R and loads of RAM. see thread: http://www.nabble.com/Error-in-matrix:--too-many-elements-specified-td20457910.html I know there are some packages on cran which help with large datasets, but i haven't got around to using them myself (yet). from: http://cran.r-project.org/web/views/HighPerformanceComputing.html [quote start...] Large memory and out-of-memory data * The bigmemory package by Kane and Emerson permits storing large objects such as matrices in memory and uses external pointer objects to refer to them. This permits transparent access from R without bumping against R's internal memory limits. Several R processes on the same computer can also shared big memory objects. * A large number of database packages, and database-alike packages (such as sqldf by Grothendieck) are also of potential interest but not (yet?) reviewed here. [quote end.] Just for fun, i just tried (for the first time): library(bigmemory) big.matrix(nrow=8, ncol=8, type='double') but i ended up crashing out of R on my low spec windows XP uni laptop :D hope that helps a little, Tony Breyal On 25 Feb, 11:04, Corrado ct...@york.ac.uk wrote: Dear friends, I have to use a very large matrix. Something of the sort of matrix(8,8,n) where n is something numeric of the sort 0.xx I have not found a way of doing it. I keep getting the error Error in matrix(nrow = 8, ncol = 8, 0.2) : too many elements specified Any suggestions? I have searched the mailing list, but to no avail. Best, -- Corrado Topi Global Climate Change Biodiversity Indicators Area 18,Department of Biology University of York, York, YO10 5YW, UK Phone: + 44 (0) 1904 328645, E-mail: ct...@york.ac.uk __ r-h...@r-project.org mailing listhttps://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guidehttp://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
