Re: [R] reading multiple files
Dave Evens wrote:
Dear All,
How do I read in multiple data frames or matrices in a
loop, e.g.
for (i in 1:n) {
channel - odbcConnectExcel(filenames)
file[i] - as.data.frame(sqlFetch(channel,
sheet))
}
I would like file[i] to be the name of the data.frame
(i.e. file[1], file[2], file[3],...etc) rather than a
vector.
The terminology you are using re. R objects seems to be rather confused.
Quite probably you want a list of data frames such as using the
filenames as names for the list elements along the follwoing lines:
library(RODBC)
dataframes - vector(mode=list, length=length(filenames))
names(dataframes) - filenames
for(i in filenames){
channel - odbcConnectExcel(i)
dataframes[[i]] - sqlFetch(channel, sheet)
close(channel)
}
Uwe Ligges
Thanks in advance for any help.
Dave
__
R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
__
R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
RE: [R] reading multiple files
You're almost there, use a list:
myfiles - list()
for (i in 1:n) myfiles[[i]] - etc
You can then get at your data frames with myfiles[[1]],
myfiles[[2]]... Or, if you prefer to combine them into a single data
frame (assuming they're similar),
allmyfiles - do.call(rbind,myfiles)
-Original Message-
From: Dave Evens [mailto:[EMAIL PROTECTED]
Sent: Tuesday, May 24, 2005 11:10 AM
To: r-help@stat.math.ethz.ch
Subject: [R] reading multiple files
Dear All,
How do I read in multiple data frames or matrices in a
loop, e.g.
for (i in 1:n) {
channel - odbcConnectExcel(filenames)
file[i] - as.data.frame(sqlFetch(channel,
sheet))
}
I would like file[i] to be the name of the data.frame
(i.e. file[1], file[2], file[3],...etc) rather than a
vector.
Thanks in advance for any help.
Dave
__
R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
__
R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Re: [R] Reading multiple files into R
On Sun, 3 Oct 2004, Vikas Rawal wrote: Thanks Kevin and Roger. This gave me the clue and was a great help. I have been trying it out. There is some problem in the code that still needs to be figured out. For the first 9 files, paste(wb-0, i, vc.dbf, sep=) works fine. But as you rightly guessed, I have more files. So when I use paste(wb-, formatC(i, width=2, flag=0), vc.dbf, sep=), dbf.read does not work. formatC works find if I use it in cat(paste.). It displays the file names correctly. But when I use it in dbf.read, it gives the following error. *** res[[i]] - maptools:::dbf.read(paste(wb-, format(i, width=2,flag=0), vc.dbf, sep=)) Error in maptools:::dbf.read(paste(wb-, format(i, width=2,flag=0), vc.dbf, sep=)) Error in maptools:::dbf.read(paste(wb-, format(i, width = 2, flag = 0), : unable to open DBF file *** Of course, the data files are all right. I can read them individually. What do you think could be the problem? Look for the difference between: for(i in 1:12) cat(paste(wb-, format(i, width=2,flag=0),vc.dbf, + sep=), \n) and for(i in 1:12) cat(paste(wb-, formatC(i, width=2,flag=0),vc.dbf, + sep=), \n) You need formatC(), not format(). sprintf() is: for(i in 1:12) cat(paste(wb-, sprintf(fmt=%0.2d, i),vc.dbf, + sep=), \n) for the same as formatC(). On the rbind question: Now I have a vector of lists res. I would like to append all these components into one single dataframe. I tried the following: rbind(for (i in 1:17) res[[i]]) - distvc But this will not work. It works if I individually specify all the res components. this works: xx - list(df1=data.frame(x=rnorm(10), y=rnorm(10), f=rep(A, 10)), + df2=data.frame(x=rnorm(10), y=rnorm(10), f=rep(B, 10))) xxx - NULL for(i in 1:length(xx)) xxx - rbind(xxx, xx[[i]]) for() loops are not a bad thing if you are not repeating the operation (like reading in data) very frequently, and seem to me easier to debug than more sophisticated constructions. This for() loop will run slower as xxx grows, because it needs to re-allocate memory each time round. I would be tempted for many and large xx[[i]] to pre-allocate the combined data frame and just slot in the rows for each list component, if I knew that the numbers and classes og the columns were identical. But rbind() is cleaner, even though it will be slower - again, if you only need this a few times, the time hit is compensated for by simplicity. Vikas Kevin Bartz wrote: Roger Bivand wrote: On Fri, 1 Oct 2004, Vikas Rawal wrote: I want to read data from a number of files into R. Reading individual files one by one requires writing enormous amount of code that will look something like the following. maptools:::dbf.read(wb-01vc.dbf)-dist1 maptools:::dbf.read(wb-02vc.dbf)-dist2 maptools:::dbf.read(wb-03vc.dbf)-dist3 maptools:::dbf.read(wb-04vc.dbf)-dist4 maptools:::dbf.read(wb-05vc.dbf)-dist5 maptools:::dbf.read(wb-06vc.dbf)-dist6 maptools:::dbf.read(wb-07vc.dbf)-dist7 maptools:::dbf.read(wb-08vc.dbf)-dist8 maptools:::dbf.read(wb-09vc.dbf)-dist9 * In this case, you could pre-allocate a list and: res - vector(mode=list, length=9) for (i in 1:length(res)) res[[i]] - maptools:::dbf.read(paste(wb-0, i, vc.dbf, sep=)) res - vector(mode=list, length=9) for (i in 1:length(res)) cat(paste(wb-0, i, vc.dbf, sep=), \n) wb-01vc.dbf wb-02vc.dbf wb-03vc.dbf ... gives a check on what file names are being used. For 10 to 99 preserving the 01-09, use paste(wb-, formatC(i, width=2, flag=0), vc.dbf, sep=). If the token is a character (string) that varies, you can roll out a character vector of tokens first and step along it. res - vector(mode=list, length=length(LETTERS)) for (i in 1:length(res)) cat(paste(wb-, LETTERS[i], vc.dbf, sep=), + \n) wb-Avc.dbf wb-Bvc.dbf wb-Cvc.dbf ... Is there a better way of doing this? Vikas __ [EMAIL PROTECTED] mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html Good call. Here's a somewhat more R-ified version: res - lapply(paste(wb-, formatC(1:99, width=2, flag=0), vc.dbf, sep=), maptools:::dbf.read) Kevin -- Roger Bivand Economic Geography Section, Department of Economics, Norwegian School of Economics and Business Administration, Breiviksveien 40, N-5045 Bergen, Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93 e-mail: [EMAIL PROTECTED] __ [EMAIL PROTECTED] mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Re: [R] Reading multiple files into R
Thanks Kevin and Roger. This gave me the clue and was a great help. I have been trying it out. There is some problem in the code that still needs to be figured out. For the first 9 files, paste(wb-0, i, vc.dbf, sep=) works fine. But as you rightly guessed, I have more files. So when I use paste(wb-, formatC(i, width=2, flag=0), vc.dbf, sep=), dbf.read does not work. formatC works find if I use it in cat(paste.). It displays the file names correctly. But when I use it in dbf.read, it gives the following error. *** res[[i]] - maptools:::dbf.read(paste(wb-, format(i, width=2,flag=0), vc.dbf, sep=)) Error in maptools:::dbf.read(paste(wb-, format(i, width=2,flag=0), vc.dbf, sep=)) Error in maptools:::dbf.read(paste(wb-, format(i, width = 2, flag = 0), : unable to open DBF file *** Of course, the data files are all right. I can read them individually. What do you think could be the problem? Vikas Kevin Bartz wrote: Roger Bivand wrote: On Fri, 1 Oct 2004, Vikas Rawal wrote: I want to read data from a number of files into R. Reading individual files one by one requires writing enormous amount of code that will look something like the following. maptools:::dbf.read(wb-01vc.dbf)-dist1 maptools:::dbf.read(wb-02vc.dbf)-dist2 maptools:::dbf.read(wb-03vc.dbf)-dist3 maptools:::dbf.read(wb-04vc.dbf)-dist4 maptools:::dbf.read(wb-05vc.dbf)-dist5 maptools:::dbf.read(wb-06vc.dbf)-dist6 maptools:::dbf.read(wb-07vc.dbf)-dist7 maptools:::dbf.read(wb-08vc.dbf)-dist8 maptools:::dbf.read(wb-09vc.dbf)-dist9 * In this case, you could pre-allocate a list and: res - vector(mode=list, length=9) for (i in 1:length(res)) res[[i]] - maptools:::dbf.read(paste(wb-0, i, vc.dbf, sep=)) res - vector(mode=list, length=9) for (i in 1:length(res)) cat(paste(wb-0, i, vc.dbf, sep=), \n) wb-01vc.dbf wb-02vc.dbf wb-03vc.dbf ... gives a check on what file names are being used. For 10 to 99 preserving the 01-09, use paste(wb-, formatC(i, width=2, flag=0), vc.dbf, sep=). If the token is a character (string) that varies, you can roll out a character vector of tokens first and step along it. res - vector(mode=list, length=length(LETTERS)) for (i in 1:length(res)) cat(paste(wb-, LETTERS[i], vc.dbf, sep=), + \n) wb-Avc.dbf wb-Bvc.dbf wb-Cvc.dbf ... Is there a better way of doing this? Vikas __ [EMAIL PROTECTED] mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html Good call. Here's a somewhat more R-ified version: res - lapply(paste(wb-, formatC(1:99, width=2, flag=0), vc.dbf, sep=), maptools:::dbf.read) Kevin __ [EMAIL PROTECTED] mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Re: [R] Reading multiple files into R
I progressed when I combined the Kevin-Roger method with the
Jomes-holtman method. sprintf() in place of formatC did the trick.
Holtman's method does not work because of some problem with the assign.
It seems you cannot have a variable target of the assignment.
Now I have a vector of lists res. I would like to append all these
components into one single dataframe.
I tried the following:
rbind(for (i in 1:17) res[[i]]) - distvc
But this will not work. It works if I individually specify all the res
components.
Vikas
***
Holtman's method:
for (i in 1:30){
assign(paste('dist', i, sep=''),
maptools:::dbf.read(sprintf(wb-%02dvc.dbf,i)))
}
Roger's method
res - vector(mode=list, length=9)
for (i in 1:length(res)) res[[i]] -
maptools:::dbf.read(paste(wb-0, i, vc.dbf, sep=))
res - vector(mode=list, length=9)
for (i in 1:length(res)) cat(paste(wb-0, i, vc.dbf, sep=), \n)
wb-01vc.dbf wb-02vc.dbf wb-03vc.dbf ...
gives a check on what file names are being used.
For 10 to 99 preserving the 01-09, use paste(wb-, formatC(i,
width=2, flag=0), vc.dbf, sep=).
If the token is a character (string) that varies, you can roll out a
character vector of tokens first and step along it.
res - vector(mode=list, length=length(LETTERS))
for (i in 1:length(res)) cat(paste(wb-, LETTERS[i], vc.dbf,
sep=),
+ \n)
wb-Avc.dbf wb-Bvc.dbf wb-Cvc.dbf ...
__
[EMAIL PROTECTED] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Re: [R] Reading multiple files into R
Hi!
There is a function ?dir which returns you the content of the dir_ectory.
If this is more then there is a function ?grep which allows you to
extract relevant items.
If you need to postprocess the names you have a function ?paste for example.
And finally you have an S language construct for(){}
And there is help.search() and An Introduction to R to which tells you
how to write functions.
/E
Vikas Rawal wrote:
I want to read data from a number of files into R.
Reading individual files one by one requires writing enormous amount
of code that will look something like the following.
maptools:::dbf.read(wb-01vc.dbf)-dist1
maptools:::dbf.read(wb-02vc.dbf)-dist2
maptools:::dbf.read(wb-03vc.dbf)-dist3
maptools:::dbf.read(wb-04vc.dbf)-dist4
maptools:::dbf.read(wb-05vc.dbf)-dist5
maptools:::dbf.read(wb-06vc.dbf)-dist6
maptools:::dbf.read(wb-07vc.dbf)-dist7
maptools:::dbf.read(wb-08vc.dbf)-dist8
maptools:::dbf.read(wb-09vc.dbf)-dist9
*
Is there a better way of doing this?
Vikas
__
[EMAIL PROTECTED] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide!
http://www.R-project.org/posting-guide.html
--
Dipl. bio-chem. Witold Eryk Wolski
MPI-Moleculare Genetic
Ihnestrasse 63-73 14195 Berlin _
tel: 0049-30-83875219 'v'
http://www.molgen.mpg.de/~wolski / \
mail: [EMAIL PROTECTED] ---W-W
[EMAIL PROTECTED]
__
[EMAIL PROTECTED] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Re: [R] Reading multiple files into R
On Fri, 1 Oct 2004, Vikas Rawal wrote: I want to read data from a number of files into R. Reading individual files one by one requires writing enormous amount of code that will look something like the following. maptools:::dbf.read(wb-01vc.dbf)-dist1 maptools:::dbf.read(wb-02vc.dbf)-dist2 maptools:::dbf.read(wb-03vc.dbf)-dist3 maptools:::dbf.read(wb-04vc.dbf)-dist4 maptools:::dbf.read(wb-05vc.dbf)-dist5 maptools:::dbf.read(wb-06vc.dbf)-dist6 maptools:::dbf.read(wb-07vc.dbf)-dist7 maptools:::dbf.read(wb-08vc.dbf)-dist8 maptools:::dbf.read(wb-09vc.dbf)-dist9 * In this case, you could pre-allocate a list and: res - vector(mode=list, length=9) for (i in 1:length(res)) res[[i]] - maptools:::dbf.read(paste(wb-0, i, vc.dbf, sep=)) res - vector(mode=list, length=9) for (i in 1:length(res)) cat(paste(wb-0, i, vc.dbf, sep=), \n) wb-01vc.dbf wb-02vc.dbf wb-03vc.dbf ... gives a check on what file names are being used. For 10 to 99 preserving the 01-09, use paste(wb-, formatC(i, width=2, flag=0), vc.dbf, sep=). If the token is a character (string) that varies, you can roll out a character vector of tokens first and step along it. res - vector(mode=list, length=length(LETTERS)) for (i in 1:length(res)) cat(paste(wb-, LETTERS[i], vc.dbf, sep=), + \n) wb-Avc.dbf wb-Bvc.dbf wb-Cvc.dbf ... Is there a better way of doing this? Vikas __ [EMAIL PROTECTED] mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html -- Roger Bivand Economic Geography Section, Department of Economics, Norwegian School of Economics and Business Administration, Breiviksveien 40, N-5045 Bergen, Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93 e-mail: [EMAIL PROTECTED] __ [EMAIL PROTECTED] mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Re: [R] Reading multiple files into R
Hoi Vikas,
--On vrijdag 1 oktober 2004 10:50 +0530 Vikas Rawal [EMAIL PROTECTED]
wrote:
I want to read data from a number of files into R.
Reading individual files one by one requires writing enormous amount of
code that will look something like the following.
Is there a better way of doing this?
These days I'm using the code below to read in each datafile I have, and
come out with a single dataframe.
# Concatenate the raw data files.
(datafiles - list.files(path=../raw data/, pattern=pp.+\.dat$))
tst - do.call('rbind', lapply(datafiles, function(x) read.table(
paste('../raw data/', x, sep=), skip=1)))
rm(datafiles)
--
Paul Lemmens
NICI, University of Nijmegen ASCII Ribbon Campaign /\
Montessorilaan 3 (B.01.05)Against HTML Mail \ /
NL-6525 HR Nijmegen X
The Netherlands / \
Phonenumber+31-24-3612648
Fax+31-24-3616066
__
[EMAIL PROTECTED] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Re: [R] Reading multiple files into R
Roger Bivand wrote: On Fri, 1 Oct 2004, Vikas Rawal wrote: I want to read data from a number of files into R. Reading individual files one by one requires writing enormous amount of code that will look something like the following. maptools:::dbf.read(wb-01vc.dbf)-dist1 maptools:::dbf.read(wb-02vc.dbf)-dist2 maptools:::dbf.read(wb-03vc.dbf)-dist3 maptools:::dbf.read(wb-04vc.dbf)-dist4 maptools:::dbf.read(wb-05vc.dbf)-dist5 maptools:::dbf.read(wb-06vc.dbf)-dist6 maptools:::dbf.read(wb-07vc.dbf)-dist7 maptools:::dbf.read(wb-08vc.dbf)-dist8 maptools:::dbf.read(wb-09vc.dbf)-dist9 * In this case, you could pre-allocate a list and: res - vector(mode=list, length=9) for (i in 1:length(res)) res[[i]] - maptools:::dbf.read(paste(wb-0, i, vc.dbf, sep=)) res - vector(mode=list, length=9) for (i in 1:length(res)) cat(paste(wb-0, i, vc.dbf, sep=), \n) wb-01vc.dbf wb-02vc.dbf wb-03vc.dbf ... gives a check on what file names are being used. For 10 to 99 preserving the 01-09, use paste(wb-, formatC(i, width=2, flag=0), vc.dbf, sep=). If the token is a character (string) that varies, you can roll out a character vector of tokens first and step along it. res - vector(mode=list, length=length(LETTERS)) for (i in 1:length(res)) cat(paste(wb-, LETTERS[i], vc.dbf, sep=), + \n) wb-Avc.dbf wb-Bvc.dbf wb-Cvc.dbf ... Is there a better way of doing this? Vikas __ [EMAIL PROTECTED] mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html Good call. Here's a somewhat more R-ified version: res - lapply(paste(wb-, formatC(1:99, width=2, flag=0), vc.dbf, sep=), maptools:::dbf.read) Kevin __ [EMAIL PROTECTED] mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
