Have you tried 'save' to save the output. 'write' is probably spending a lot of time converting to character. Are you just going to read it back into R for processing; if so 'save' will probably be faster.
On Mon, Mar 26, 2012 at 12:53 PM, Diann Prosser <[email protected]> wrote: > Dear all, > I am working with large matrices (19.6 million elements * 1000 simulations) > and am trying to get around memory problems and vector length issues. I’ve > split the inputs so that the output vector length will not exceed 2^31. > Working on a 64bit machine with 80GB RAM, I still get close to the memory > limits when allowing output to be held in memory (as would be expected). > > Originally, I planned to write the output to a file after it was produced in > memory. It took 15min for the output to be produced, but now it's been > working on writing it to a file for almost an hour (and ongoing). > > Is the recommended way to manage large output like this to write it directly > to a file? > Can that be done as the output is produced, so that memory usage does not > build (i.e., so it’s not storing it in memory)? > Is that what Sink is designed to do? I’ve been trying to find information > about this on the help archive as well as the R > http://stat.ethz.ch/R-manual/R-devel/doc/manual/R-data.html Data > Import/Export Manual but it’s still not clear to me. > > Many thanks for your guidance. > > Some more info: >>dim(stPte801) > NULL #it's a vector >> length(stPte801) > [1] 1965705000 >>write(stPte801, "stPte801.txt", sep="\n") > #it's been writing for almost an hour... > #eventually, I will need to pull it back into R to do the next step (but > after the other variables are created) > > -- > View this message in context: > http://r.789695.n4.nabble.com/writing-output-directly-to-file-sink-tp4506432p4506432.html > Sent from the R help mailing list archive at Nabble.com. > > ______________________________________________ > [email protected] mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. -- Jim Holtman Data Munger Guru What is the problem that you are trying to solve? Tell me what you want to do, not how you want to do it. ______________________________________________ [email protected] mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
