Hi,
On Fri, Feb 3, 2012 at 1:12 PM, HC hca...@yahoo.co.in wrote:
Thank you.
The readLines command is working fine and I am able to read 10^6 lines in
one go and write them using the write.table command.
Does this readLines command using a block concept to optimize or goes line
by line?
Thank you Jim for your reply.
I could figure out that readLines works fine until 35,841,335 lines
(records).
When the next line is read to be read, a window with R for Windows GUI
front-end has stopped working message comes, with an option to close
program or checking online for a solution.
Thank you for indicating that SQLite may not handle a file as big as 160 GB.
Would you know of any utility for *physically splitting *the 160 GB text
file into pieces. And if one can control the splitting at the end of a
record.
Thank you again.
HC
--
View this message in context:
On Fri, Feb 3, 2012 at 6:03 AM, HC hca...@yahoo.co.in wrote:
Thank you for indicating that SQLite may not handle a file as big as 160 GB.
Would you know of any utility for *physically splitting *the 160 GB text
file into pieces. And if one can control the splitting at the end of a
record.
On Fri, Feb 3, 2012 at 7:37 AM, Gabor Grothendieck
ggrothendi...@gmail.com wrote:
On Fri, Feb 3, 2012 at 6:03 AM, HC hca...@yahoo.co.in wrote:
Thank you for indicating that SQLite may not handle a file as big as 160 GB.
Would you know of any utility for *physically splitting *the 160 GB text
This is a 160 GB tab-separated .txt file. It has 9 columns and 3.25x10^9
rows.
Can R handle it?
Thank you.
HC
--
View this message in context:
http://r.789695.n4.nabble.com/sqldf-for-Very-Large-Tab-Delimited-Files-tp4350555p4354556.html
Sent from the R help mailing list archive at
On Fri, Feb 3, 2012 at 8:08 AM, HC hca...@yahoo.co.in wrote:
This is a 160 GB tab-separated .txt file. It has 9 columns and 3.25x10^9
rows.
Can R handle it?
You can process a file N lines at time like this:
con - file(myfile.dat, r)
while(length(Lines - readLines(con, n = N)) 0) {
...
Thank you.
The readLines command is working fine and I am able to read 10^6 lines in
one go and write them using the write.table command.
Does this readLines command using a block concept to optimize or goes line
by line?
Steve has mentioned about *nix and split commands. Would there be any
Bad news!
The readLines command works fine upto a certain limit. Once a few files have
been written the R program crashes.
I used the following code:
*
iFile-Test.txt
con - file(iFile, r)
N-125;
iLoop-1
while(length(Lines - readLines(con, n = N)) 0 iLoop41) {
Exactly what does crashed mean? What was the error message? How
you tried to put:
rm(Lines)
gc()
at the end of the loop to free up and compact memory? If you watch
the performance, does the R process seem to be growing in terms of the
amount of memory that is being used? You can add:
On Wed, Feb 1, 2012 at 11:57 PM, HC hca...@yahoo.co.in wrote:
Hi All,
I have a very (very) large tab-delimited text file without headers. There
are only 8 columns and millions of rows. I want to make numerous pieces of
this file by sub-setting it for individual stations. Station is given as
On Thu, Feb 2, 2012 at 3:11 AM, Gabor Grothendieck
ggrothendi...@gmail.com wrote:
On Wed, Feb 1, 2012 at 11:57 PM, HC hca...@yahoo.co.in wrote:
Hi All,
I have a very (very) large tab-delimited text file without headers. There
are only 8 columns and millions of rows. I want to make numerous
Hi Gabor,
Thank you very much for your guidance and help.
I could run the following code successfully on a 500 mb test data file. A
snapshot of the data file is attached herewith.
code start***
library(sqldf)
library(RSQLite)
iFile-Test100.txt
con -
On Thu, Feb 2, 2012 at 8:07 PM, HC hca...@yahoo.co.in wrote:
Hi Gabor,
Thank you very much for your guidance and help.
I could run the following code successfully on a 500 mb test data file. A
snapshot of the data file is attached herewith.
code start***
library(sqldf)
Hi All,
I have a very (very) large tab-delimited text file without headers. There
are only 8 columns and millions of rows. I want to make numerous pieces of
this file by sub-setting it for individual stations. Station is given as in
the first column. I am trying to learn and use sqldf package for
15 matches
Mail list logo
