[R] How to signal the end of the table?
I am using a for loop to read a table row by row and I have to specify how
many records are there in the table. I need to read row by row because the
table is huge and the memory not large enough for the whole table.:
number.of.records=100
fp=file(abc.csv,r)
pos=seek(fp, rw=read)
for (i in 1:number.of.record){
current.row=scan(file=fp, sep=',', what=list(count=1, cusip6=, idate=1,
spread=1.1, vol252=1.1, vol1000=1.1, st_debt=1.1, lt_debt=1.1 , total_liab=
1.1, cr=1.1, shrout=1.1, prc=1.1, mkt_cap=1.1, rtng=1.1, sec=1.1, cr3m=1.1,
cr5y=1.1, ust3m=1.1, ust5y=1.1), flush=TRUE, nlines=1,quiet=T)
...
}
I need to know the number of records in the table and put it in the variable
named number.of.records. When I have a new table that I do not know how
many records it has, I use excel to open the file to figure it out and put
it in variable number.of.records. I often have many tables to try and
every one of them has thousands of recordsit takes a lot of time and
trouble to adjust the code every time I read a different table.
I am wondering if I can change the for loop to a while loop:
while (the end of the table has not been reached)
{
current.row=scan(file=fp, sep=',', what=list(count=1, cusip6=, idate=1,
spread=1.1, vol252=1.1, vol1000=1.1, st_debt=1.1, lt_debt=1.1 , total_liab=
1.1, cr=1.1, shrout=1.1, prc=1.1, mkt_cap=1.1, rtng=1.1, sec=1.1, cr3m=1.1,
cr5y=1.1, ust3m=1.1, ust5y=1.1), flush=TRUE, nlines=1,quiet=T)
...
}
The problem is how to articulate while (the end of the table has not been
reached), or equivalently, how to signal the end of the table?
Best Wishes
Yuchen Luo
[[alternative HTML version deleted]]
__
R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] How to signal the end of the table?
What about trying to use a database system and make queries?
If you are at the end of a file, you will read elements of length 0...
Uwe Ligges
Yuchen Luo wrote:
I am using a for loop to read a table row by row and I have to specify how
many records are there in the table. I need to read row by row because the
table is huge and the memory not large enough for the whole table.:
number.of.records=100
fp=file(abc.csv,r)
pos=seek(fp, rw=read)
for (i in 1:number.of.record){
current.row=scan(file=fp, sep=',', what=list(count=1, cusip6=, idate=1,
spread=1.1, vol252=1.1, vol1000=1.1, st_debt=1.1, lt_debt=1.1 , total_liab=
1.1, cr=1.1, shrout=1.1, prc=1.1, mkt_cap=1.1, rtng=1.1, sec=1.1, cr3m=1.1,
cr5y=1.1, ust3m=1.1, ust5y=1.1), flush=TRUE, nlines=1,quiet=T)
...
}
I need to know the number of records in the table and put it in the variable
named number.of.records. When I have a new table that I do not know how
many records it has, I use excel to open the file to figure it out and put
it in variable number.of.records. I often have many tables to try and
every one of them has thousands of recordsit takes a lot of time and
trouble to adjust the code every time I read a different table.
I am wondering if I can change the for loop to a while loop:
while (the end of the table has not been reached)
{
current.row=scan(file=fp, sep=',', what=list(count=1, cusip6=, idate=1,
spread=1.1, vol252=1.1, vol1000=1.1, st_debt=1.1, lt_debt=1.1 , total_liab=
1.1, cr=1.1, shrout=1.1, prc=1.1, mkt_cap=1.1, rtng=1.1, sec=1.1, cr3m=1.1,
cr5y=1.1, ust3m=1.1, ust5y=1.1), flush=TRUE, nlines=1,quiet=T)
...
}
The problem is how to articulate while (the end of the table has not been
reached), or equivalently, how to signal the end of the table?
Best Wishes
Yuchen Luo
[[alternative HTML version deleted]]
__
R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
__
R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] How to signal the end of the table?
Have you tried read.table() and the similar functions? I think they
would provide a much more simple solution for your problem.
HTH,
Thierry
ir. Thierry Onkelinx
Instituut voor natuur- en bosonderzoek / Research Institute for Nature
and Forest
Cel biometrie, methodologie en kwaliteitszorg / Section biometrics,
methodology and quality assurance
Gaverstraat 4
9500 Geraardsbergen
Belgium
tel. + 32 54/436 185
[EMAIL PROTECTED]
www.inbo.be
Do not put your faith in what statistics say until you have carefully
considered what they do not say. ~William W. Watt
A statistical analysis, properly conducted, is a delicate dissection of
uncertainties, a surgery of suppositions. ~M.J.Moroney
-Oorspronkelijk bericht-
Van: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] Namens Yuchen Luo
Verzonden: woensdag 29 augustus 2007 19:41
Aan: r-help@stat.math.ethz.ch
Onderwerp: [R] How to signal the end of the table?
I am using a for loop to read a table row by row and I have
to specify how many records are there in the table. I need to
read row by row because the table is huge and the memory not
large enough for the whole table.:
number.of.records=100
fp=file(abc.csv,r)
pos=seek(fp, rw=read)
for (i in 1:number.of.record){
current.row=scan(file=fp, sep=',', what=list(count=1,
cusip6=, idate=1, spread=1.1, vol252=1.1, vol1000=1.1,
st_debt=1.1, lt_debt=1.1 , total_liab=
1.1, cr=1.1, shrout=1.1, prc=1.1, mkt_cap=1.1, rtng=1.1,
sec=1.1, cr3m=1.1,
cr5y=1.1, ust3m=1.1, ust5y=1.1), flush=TRUE, nlines=1,quiet=T)
...
}
I need to know the number of records in the table and put it
in the variable named number.of.records. When I have a new
table that I do not know how many records it has, I use excel
to open the file to figure it out and put it in variable
number.of.records. I often have many tables to try and every
one of them has thousands of recordsit takes a lot of
time and trouble to adjust the code every time I read a
different table.
I am wondering if I can change the for loop to a while loop:
while (the end of the table has not been reached)
{
current.row=scan(file=fp, sep=',', what=list(count=1,
cusip6=, idate=1, spread=1.1, vol252=1.1, vol1000=1.1,
st_debt=1.1, lt_debt=1.1 , total_liab=
1.1, cr=1.1, shrout=1.1, prc=1.1, mkt_cap=1.1, rtng=1.1,
sec=1.1, cr3m=1.1,
cr5y=1.1, ust3m=1.1, ust5y=1.1), flush=TRUE, nlines=1,quiet=T)
...
}
The problem is how to articulate while (the end of the table
has not been reached), or equivalently, how to signal the
end of the table?
Best Wishes
Yuchen Luo
[[alternative HTML version deleted]]
__
R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
__
R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
[R] How to signal the end of the table?
Well, i am surprise you have problems to read a table that is small enough to
be opened entirely in Excel.
I work with csv tables with hundreds of thousands of rows, and sometimes even
millions with no problems except that sometimes i have to wait up to 1 or
2 minutes for R to read the table. I use the command read.csv.
If your tables are always small enough to be read in Excel, the limit of number
of rows in Excel is about 64000 or something of that sort (maybe 65000 ) so
you can use this number as a limit for your number of rows although you
may have to clean-up your data.frame afterwards.
I work on a Windows machine with 4 Gb DRAM - just for comparison.
I hope this helps,
Monica
Message: 88Date: Wed, 29 Aug 2007 10:41:05 -0700From: Yuchen Luo [EMAIL
PROTECTED]Subject: [R] How to signal the end of the table?To:
r-help@stat.math.ethz.chMessage-ID:[EMAIL PROTECTED]Content-Type:
text/plainI am using a for loop to read a table row by row and I have to
specify howmany records are there in the table. I need to read row by row
because thetable is huge and the memory not large enough for the whole
table.:number.of.records=100fp=file(abc.csv,r)pos=seek(fp,
rw=read)for (i in 1:number.of.record){current.row=scan(file=fp, sep=',',
what=list(count=1, cusip6=, idate=1,spread=1.1, vol252=1.1, vol1000=1.1,
st_debt=1.1, lt_debt=1.1 , total_liab=1.1, cr=1.1, shrout=1.1, prc=1.1,
mkt_cap=1.1, rtng=1.1, sec=1.1, cr3m=1.1,cr5y=1.1, ust3m=1.1, ust5y=1.1),
flush=TRUE, nlines=1,quiet=T)...}I need to know the number of records
in the table and put it in the variablenamed nu!
mber.of.records. When I have a new table that I do not know howmany records
it has, I use excel to open the file to figure it out and putit in variable
number.of.records. I often have many tables to try andevery one of them has
thousands of recordsit takes a lot of time andtrouble to adjust the code
every time I read a different table.I am wondering if I can change the
for loop to a while loop:while (the end of the table has not been
reached){current.row=scan(file=fp, sep=',', what=list(count=1, cusip6=,
idate=1,spread=1.1, vol252=1.1, vol1000=1.1, st_debt=1.1, lt_debt=1.1 ,
total_liab=1.1, cr=1.1, shrout=1.1, prc=1.1, mkt_cap=1.1, rtng=1.1, sec=1.1,
cr3m=1.1,cr5y=1.1, ust3m=1.1, ust5y=1.1), flush=TRUE,
nlines=1,quiet=T)...}The problem is how to articulate while (the end of
the table has not beenreached), or equivalently, how to signal the end of the
table?Best WishesYuchen Luo [[alternative HTML version deleted]]
_
Discover the new Windows Vista
[[alternative HTML version deleted]]
__
R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] How to signal the end of the table?
For example,
fp-file(abc.csv,r)
c.row-scan(file=fp,sep=,,nlines=1) # what argument is omitted for
bevity as it doesn't matter
rows-c.row
while(length(c.row)0) {
c.row-scan(file=fp,sep=;,nlines=1);
rows-rbind(rows,c.row)
}
close(fp)
If you want to read a file by parts, then you do something like the
following (untested)
But this looks like a C-style approach, not R-style
fp-file(abc.csv,r)
lines.skip-0
for(i in 1:nparts){
c.row-scan(file=fp,sep=,,nlines=1,skip=lines.skip)
lines.read-1
rows-c.row
while(length(c.row)0 lines.readpart.size) {
c.row-scan(file=fp,sep=;,nlines=1); # nlines can be also 1
rows-rbind(rows,c.row)
lines.read-lines.read+1 # if nlines above is 1 then +1 must be
replaced with nlines value
}
lines.skip-lines.read
# do operations with rows
}
close(fp)
Yuchen Luo wrote:
I am using a for loop to read a table row by row and I have to specify
how
many records are there in the table. I need to read row by row because the
table is huge and the memory not large enough for the whole table.:
number.of.records=100
fp=file(abc.csv,r)
pos=seek(fp, rw=read)
for (i in 1:number.of.record){
current.row=scan(file=fp, sep=',', what=list(count=1, cusip6=, idate=1,
spread=1.1, vol252=1.1, vol1000=1.1, st_debt=1.1, lt_debt=1.1 ,
total_liab=
1.1, cr=1.1, shrout=1.1, prc=1.1, mkt_cap=1.1, rtng=1.1, sec=1.1,
cr3m=1.1,
cr5y=1.1, ust3m=1.1, ust5y=1.1), flush=TRUE, nlines=1,quiet=T)
...
}
I need to know the number of records in the table and put it in the
variable
named number.of.records. When I have a new table that I do not know how
many records it has, I use excel to open the file to figure it out and put
it in variable number.of.records. I often have many tables to try and
every one of them has thousands of recordsit takes a lot of time and
trouble to adjust the code every time I read a different table.
I am wondering if I can change the for loop to a while loop:
while (the end of the table has not been reached)
{
current.row=scan(file=fp, sep=',', what=list(count=1, cusip6=, idate=1,
spread=1.1, vol252=1.1, vol1000=1.1, st_debt=1.1, lt_debt=1.1 ,
total_liab=
1.1, cr=1.1, shrout=1.1, prc=1.1, mkt_cap=1.1, rtng=1.1, sec=1.1,
cr3m=1.1,
cr5y=1.1, ust3m=1.1, ust5y=1.1), flush=TRUE, nlines=1,quiet=T)
...
}
The problem is how to articulate while (the end of the table has not been
reached), or equivalently, how to signal the end of the table?
Best Wishes
Yuchen Luo
--
View this message in context:
http://www.nabble.com/How-to-signal-the-end-of-the-table--tf4353030.html#a12408586
Sent from the R help mailing list archive at Nabble.com.
__
R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
