On Thu, 23 Aug 2007, Yupu Liang wrote: > Dear Rs: > > Hi, I am trying to read a big text file (nrows=243440, ncols=144). It > seems the computational time of all the read methods > (scan,readtable,read.delim) is not linear to the number of rows I > want to read in: things became really slow once I tried to read in > 100000 lines compare to 10000 lines).
What did 'top' or gc() tell you about memory use, paging, and swap behavior? Likely you are starting to swap when you get to nrows=100000. Even if your data were all numeric, over 100MB would be needed just to store them. And reading them in requires more memory still. > > If I am reading the profiling result right, I guess scan wouldn't > help either. > > My questions are : > 1) Is this a memory issue? > 2) How to get around this?: I can't just sit around for 15 mins. > Would write a c function help? > > Thanks! > > Here is the profiling I did: > > > Rprof() > > dd = read.delim(file,skip=9,sep="\t",as.is= T,nrows=10000) > > Rprof(NULL) > > summaryRprof() > $by.self > self.time self.pct total.time total.pct > "scan" 3.56 85.2 3.56 85.2 > "type.convert" 0.48 11.5 0.48 11.5 > "read.table" 0.08 1.9 4.18 100.0 > "make.names" 0.02 0.5 0.02 0.5 > "options" 0.02 0.5 0.02 0.5 > "readLines" 0.02 0.5 0.02 0.5 > "read.delim" 0.00 0.0 4.18 100.0 > "file" 0.00 0.0 0.02 0.5 > "getOption" 0.00 0.0 0.02 0.5 > > $by.total > total.time total.pct self.time self.pct > "read.table" 4.18 100.0 0.08 1.9 > "read.delim" 4.18 100.0 0.00 0.0 > "scan" 3.56 85.2 3.56 85.2 > "type.convert" 0.48 11.5 0.48 11.5 > "make.names" 0.02 0.5 0.02 0.5 > "options" 0.02 0.5 0.02 0.5 > "readLines" 0.02 0.5 0.02 0.5 > "file" 0.02 0.5 0.00 0.0 > "getOption" 0.02 0.5 0.00 0.0 > > $sampling.time > [1] 4.18 > > > ?Rprof() > > Rprof() > > dd = read.delim(file,skip=9,sep="\t",as.is= T,nrows=100000) > > Rprof(NULL) > > summaryRprof() > $by.self > self.time self.pct total.time total.pct > "scan" 143.12 92.7 143.12 92.7 > "type.convert" 9.52 6.2 9.52 6.2 > "read.table" 1.60 1.0 154.28 99.9 > "paste" 0.02 0.0 0.08 0.1 > "textConnection" 0.02 0.0 0.04 0.0 > ".deparseOpts" 0.02 0.0 0.02 0.0 > "file" 0.02 0.0 0.02 0.0 > "make.names" 0.02 0.0 0.02 0.0 > "print.default" 0.02 0.0 0.02 0.0 > "read.delim" 0.00 0.0 154.28 99.9 > "doTryCatch" 0.00 0.0 0.08 0.1 > "gsub" 0.00 0.0 0.08 0.1 > "try" 0.00 0.0 0.08 0.1 > "tryCatch" 0.00 0.0 0.08 0.1 > "tryCatchList" 0.00 0.0 0.08 0.1 > "tryCatchOne" 0.00 0.0 0.08 0.1 > "capture.output" 0.00 0.0 0.06 0.0 > "deparse" 0.00 0.0 0.02 0.0 > "eval.with.vis" 0.00 0.0 0.02 0.0 > "evalVis" 0.00 0.0 0.02 0.0 > "print" 0.00 0.0 0.02 0.0 > > $by.total > total.time total.pct self.time self.pct > "read.table" 154.28 99.9 1.60 1.0 > "read.delim" 154.28 99.9 0.00 0.0 > "scan" 143.12 92.7 143.12 92.7 > "type.convert" 9.52 6.2 9.52 6.2 > "paste" 0.08 0.1 0.02 0.0 > "doTryCatch" 0.08 0.1 0.00 0.0 > "gsub" 0.08 0.1 0.00 0.0 > "try" 0.08 0.1 0.00 0.0 > "tryCatch" 0.08 0.1 0.00 0.0 > "tryCatchList" 0.08 0.1 0.00 0.0 > "tryCatchOne" 0.08 0.1 0.00 0.0 > "capture.output" 0.06 0.0 0.00 0.0 > "textConnection" 0.04 0.0 0.02 0.0 > ".deparseOpts" 0.02 0.0 0.02 0.0 > "file" 0.02 0.0 0.02 0.0 > "make.names" 0.02 0.0 0.02 0.0 > "print.default" 0.02 0.0 0.02 0.0 > "deparse" 0.02 0.0 0.00 0.0 > "eval.with.vis" 0.02 0.0 0.00 0.0 > "evalVis" 0.02 0.0 0.00 0.0 > "print" 0.02 0.0 0.00 0.0 > > $sampling.time > [1] 154.36 > > I am using R 2.5.1 for mac on a Dual 2 GHz PowerPC G5 with 1GB memory. > > ______________________________________________ > R-help@stat.math.ethz.ch mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. > Charles C. Berry (858) 534-2098 Dept of Family/Preventive Medicine E mailto:[EMAIL PROTECTED] UC San Diego http://famprevmed.ucsd.edu/faculty/cberry/ La Jolla, San Diego 92093-0901 ______________________________________________ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
