Re: [R] Best way to merge 300+ .5MB dataframes?

2014-08-12 Thread David Winsemius
On Aug 11, 2014, at 8:01 PM, John McKown wrote: On Mon, Aug 11, 2014 at 9:43 PM, Thomas Adams tea...@gmail.com wrote: Grant, Assuming all your filenames are something like file1.txt, file2.txt,file3.txt... And using the Mac OSX terminal app (after you cd to the directory where your files

Re: [R] Best way to merge 300+ .5MB dataframes?

2014-08-12 Thread Prof Brian Ripley
On 12/08/2014 07:07, David Winsemius wrote: On Aug 11, 2014, at 8:01 PM, John McKown wrote: On Mon, Aug 11, 2014 at 9:43 PM, Thomas Adams tea...@gmail.com wrote: Grant, Assuming all your filenames are something like file1.txt, file2.txt,file3.txt... And using the Mac OSX terminal app (after

Re: [R] Best way to merge 300+ .5MB dataframes?

2014-08-12 Thread Grant Rettke
Thank you all kindly. Grant Rettke | ACM, AMA, COG, IEEE gret...@acm.org | http://www.wisdomandwonder.com/ “Wisdom begins in wonder.” --Socrates ((λ (x) (x x)) (λ (x) (x x))) “Life has become immeasurably better since I have been forced to stop taking it seriously.” --Thompson On Tue, Aug 12,

Re: [R] Best way to merge 300+ .5MB dataframes?

2014-08-11 Thread Grant Rettke
On Sun, Aug 10, 2014 at 6:50 PM, John McKown john.archie.mck...@gmail.com wrote: OK, I assume this results in a vector of file names in a variable, like you'd get from list.files(); Yes. Why? Do you need them in separate data frames? I do not. The meat of the question. If you don't need

Re: [R] Best way to merge 300+ .5MB dataframes?

2014-08-11 Thread Thomas Adams
Grant, Assuming all your filenames are something like file1.txt, file2.txt,file3.txt... And using the Mac OSX terminal app (after you cd to the directory where your files are located... This will strip off the 1st lines, that is, your header lines: for file in *.txt;do sed -i '1d'${file}; done

Re: [R] Best way to merge 300+ .5MB dataframes?

2014-08-11 Thread John McKown
On Mon, Aug 11, 2014 at 9:43 PM, Thomas Adams tea...@gmail.com wrote: Grant, Assuming all your filenames are something like file1.txt, file2.txt,file3.txt... And using the Mac OSX terminal app (after you cd to the directory where your files are located... This will strip off the 1st lines,

Re: [R] Best way to merge 300+ .5MB dataframes?

2014-08-10 Thread Jeff Newmiller
Just load the data frames into a list and give that list to rbind. It is way more efficient to be able to identify how big the final data frame is going to have to be at the beginning and preallocate the result memory than to incrementally allocate larger and larger data frames along the way

Re: [R] Best way to merge 300+ .5MB dataframes?

2014-08-10 Thread David Winsemius
On Aug 10, 2014, at 11:51 AM, Grant Rettke wrote: Good afternoon, Today I was working on a practice problem. It was simple, and perhaps even realistic. It looked like this: • Get a list of all the data files in a directory • Load each file into a dataframe • Merge them into a single

Re: [R] Best way to merge 300+ .5MB dataframes?

2014-08-10 Thread Jeff Newmiller
Err... sorry... you have to use do.call with base rbind as David illustrates. I am spoiled by rbind.fill from the plyr package. rbind.fill accepts the list directly and also fills in any missing columns with NA, which avoids having to dig through all the files to find any oddballs.

Re: [R] Best way to merge 300+ .5MB dataframes?

2014-08-10 Thread John McKown
On Sun, Aug 10, 2014 at 1:51 PM, Grant Rettke g...@wisdomandwonder.com wrote: Good afternoon, Today I was working on a practice problem. It was simple, and perhaps even realistic. It looked like this: • Get a list of all the data files in a directory OK, I assume this results in a vector